You'll find this problem magically disappears if you quit trying to treat classes in this game like you do classes in traditional RPGs. They're guidelines, not jobs. The system was never designed with the intention of you being an optimally allocated conjurer, then an optimally allocated gladiator, etc.