Question: What is the point of gear sets?

[Kalise]

But for real, the sets all have varying secondary stats.

But, outside of Tenacity (On Fending gear), Piety (On Healing gear) and Direct Hit (Absent from Fending/Healing gear) that's not really a reason to have different sets.

Especially when all jobs have the same stat priorities based around Crit/DH/Det/SkS/SpS with Tenacity and Piety not being wanted...

If you simply just had items with different secondary stats. People would then just pick the items with the optimal secondary stats.

Essentially, meaning that they wouldn't end up having to design say, several different Fending helmets all with different secondary stats to make getting BiS gear require getting items from multiple different sources (I.e. Raids, Tomes, Eureka, Crafting) to try and get secondary stats to align (With then mathing out whether the Primary Stat gain from a higher ilevel item is worth the loss of several hundred of a optimal stat in favour of a less optimal stat)

Someone could more easily gear up with their optimal stats via a single route. By just choosing the higher level item with stats they want on it.

While I do agree that some accessories could potentially be adjusted if there's no primary stat on them except for Fending ones, having primary stats on armor provides a more visible way for the player to increase the power of their character without always having to resort to some arcane theorycrafting min/maxing guide.

Note that I've not actually said to remove Primary Stats altogether. Just to remove the 4 different ones.

I.e. Consolidate them all down to a singular Primary Stat. Since that is essentially what they are given that they're all identical and all maxxed out with the only thing that was in any way relevant to them being different (Materia) being removed.

Alternatively, you can remove sets by simply doing an Aetheryte Earring and having items swap Primary Stat based on what job you are.

The result would being that they could continue to make Glamour sets. Such as having a bunch of Plate armour, Medium armour, Light armour and Robes.

Just that these Glamours wouldn't be restricted to certain jobs with the items themselves merely being differentiated by the secondary stats they have on them (I.e. They could still make it so that Tenacity is still found on the heavy Plate armour glamour sets and Piety is found on the Robes. Just, Tanks/Healers won't be forced to wear these items if they want an ilevel increase but don't want those stats, in favour of items that have say, Crit or Det on)

[Tharne]

The result would being that they could continue to make Glamour sets. Such as having a bunch of Plate armour, Medium armour, Light armour and Robes.

Just that these Glamours wouldn't be restricted to certain jobs

This is one of the reason. They want job restrictions on gear and they will not change that, imo.
And before the "but we can already wear this... so why not that..." : exception not the rule.