My suggestion is to keep the armory system, but add armor which gets a bonus if you have the corresponding class equiped, and if you don't you get a penalty.