That's a false equivalence. Nobody is arguing to prevent you from wearing lore-appropriate gear if you wish to do so.
In any case, glamour is just as critical to a lore-appropriate outfit as a novelty one. Is your tank looking lore-appropriate in the YoRHa dress or Shisui set? Your dancer in an outfit designed with machinist in mind? Your red mage in bulky wizard robes?
Gear can't simultaneously cater to the aesthetic of all the jobs that need to equip it, even if they didn't throw in novelty themed sets every so often.