That is because that is exactly what the armoury system was designed to do BUT it was changed before launch. Pre-launch (even before Alpha) the plan was to let Physical Level dictate you character growth. Physical Level was your Job but it was a completely blank slate. You created your own Job by adding skills and abilities from the Class system to create a truly unique Job/Character.
As Physical Level dictated your character it meant as it went up you could fight tougher things as that is what you were judged on. So if you had levelled a few classes and capped Physical Level even if you switched to a Rank 1 Class you could still fight much tougher mobs. You used affinity to help you specialise in skills that were outside of your currently equipped Class. It is only when you see this that the whole Armoury system fits into place. Classes are the building blocks to create your custom Job and Physical Level helps you shape that character in the direction you want to go.
However they changed it around and made Class Rank dominant and that has thrown the whole system out of sync really. It has turned Classes into half Jobs but as they were never designed to actually be Jobs they just feel a bit meh. Now SE are going even further away from the original vision and removing Physical Level completely, limiting cross classing further and adding a Job system on top of the remnants of the Class system and it just seems like a bad base to build on given the direction it looks like they are going.
