It is a lot of grinding for what you're getting, yes. Too much, probably, even, but this is a complaint that was already mentioned in the live letter. Their excise was to make it a similar amount of effort as the raid weapons, and with the raid being as inaccessible s it was this time, that resulted in a stupid grind.
Whether you accept that excuse or not (it's a bit half-baked, in my opinion), the fact remains that they plan on cutting down the grind (and raid difficulty) for the next tier.
Have you played any of the games you're listing?
Rift did it 'right', sure. There's so many builds that finding the 'best' ones is difficult and, usually, isn't widely known until near the end of a balance cycle. Which means there's some meaningful choices! Great, right? Well sure, except that it couldn't maintain a subscriber base. Why? Most common complaint was that it was inaccessible for most people. There were too many choices, and no elitistjerks, or whatever site/group to give people their builds for them with any kind of authority. And so it went F2P and added the option of having your build decided for you by the game.
SWtOR streamlined theirs, and they did it because the old system was a complete mess with drastic rebalances coming almost every patch--relearning how to play my character over and over again during these times is a big part of why I quit (despite the fact I was playing a class they kept buffing). And, from what I understand (I've played a few times since, but not for long), their new system is still a mess leaving certain classes extremely undertuned. It also went F2P and is clearly struggling, if you ever play it.
ESO only has a couple viable builds per class, and they are, for the most part, the difference between a tank, heal, and DPS tree. Further, it's an action mmo, akin to Tera, and can't really be compared to a straight hotbar mmo. Same with BnS.
I haven't played the others but there's one other important point to be made:
As you said, they are all niche MMOs. FFXIV is not. And there's a reason for that, and part of that reason is that it is accessible. Part of it is you don't have to carry five gear sets for every class and build macros to swap items in combat for every attack. Part of it is the fact that you don't have to relearn your entire class every month when the developers realize a couple talents synergize in an unexpected way and redo your entire talent tree. Part of it is the fact you don't have to go searching online every other level to make sure you aren't breaking your character, only to find out that your choice at this level is meaningless and will never affect your play.
Asking FFXIV to implement mechanics that main stream titles avoid, after having had them and finding they don't work well, and that only niche games have, is asking FFXIV to become a niche game. Maybe you'd like it more, but the majority of people won't.
PvE competition looks like hunts. A bunch of people fighting over limited resources with nothing really mattering other than who gets there first. Or PvE contested areas in many games where everyone ignores it and no one cares (SWtOR, WoW, and others I've played have done this and I've never, in my decades of gaming, seen anyone actually participate in these mechanics other than incidentally while moving through the areas for other reasons). PvP requires development time and rebalancing of classes to work in a PvP environment, which is never the same as PvE balance, and also trends to be for a smaller portion of the player base, as PvPers tend to gravitate toward more PvP based games.
And I actually like open world PvP. I played almost exclusively on an open PvP server in Rift (I had a PvE server character to play with family that disliked PvP). I loved it.
However, I've seen many games go to absolute garbage trying to balance the two, and either breaking one for the other, or just having an unsatisfactory experience all around. It's best to focus on one or the other.
And again, PvE competition tends to either be ignored or just frustrating with no reason to improve.
If you really want to compete go download a parser and try to have the highest DPS, or whatever, in every dungeon you run. Just don't be a dick about it to the other people in the run.