The physical level system has been altered from it's original design which is why it is not functioning properly. Originally your character would have been judged on it's Physical level and not it's class rank. What I mean by this is that if you were a rank 50 physical level you could fight mobs that were rank 50 even if your class level was rank 10. Doing this allowed you to shape your character in any way you wished and the classes would be the tools to shape and build your class any way you wished.
Once you realise this you can see how all the pieces of the armoury system fit into place by taking skills and abilities from your weapon specialisations to truly create your own character.
However for some reason that I do not know they changed it around and made your class level dictate your characters growth. This has thrown the whole armoury system out of whack so to speak. Yes you can still chop and change but your freedom is limited and watered down to how it was originally designed to work.