If there is any argument for keeping some restrictions it is for the artifact gear. A lot of people (not everyone) don't want to see an iconic White Mage robe and have that character not currently be a white mage. The other side is that if your character earned that gear why shouldn't they be able to use it?

The thing about some of these sets is that not all pieces are iconic. For White mage the robe definitely is. The pants aren't even seen and pretty much just look like swimsuit bottoms. The boots are hardly seen. and the gloves are seen but are not immediately noticed. They are pretty simple and thus could be useful in other sets without treading on the white mage look.