I actually like how it is now, i.e. that cloth wearers cannot wear heavy armor, and heavey armor jobs can't wear cloth armor.
Note that this should not be thrown into the same pile as no-restriction pure-glamour items. We can see that it's a bikini. We can see it's a chocobo suit. We can see those glamour sets. That's fine. They're in their own class. But combat armor is very different. It's all part of the coherence of the world. There's a dedicate balance to be maintained.
Think about it. I'm sure we're all fine seeing doctors wearing T-shirts and soldiers wearing beach shorts. But it starts to feel weird when doctors wear camo and soldiers are walking around in scrubs. Or when construction workers wear court dresses and athletes wear the EMU suit. Those "combat gear" has a much more emphsized effect in creating cohesion than "leisure gear".
I support job restrictions on glamour, and I don't think these restrictions should be removed across the board.