They need to make it so that spells are A LOT more efficient on the original class than they are on another class (which, if I remember, is what the affinity system is for but it's not doing too much right now). One of the best and easiest ways to do this, in my opinion, would be to lower the bases on all skills and change how they scale with stats...so that speccing into INT/MND/PIE will give you visible changes to spell damage/healing with every little bit (even with +30 in my INT, I barely get something like a 20-30 damage increase in my THM spells).

This way skills aren't restricted by use but by effectiveness.