First off, I'm sure you mean 'mathematically perfect build', because in a system like that there is no 'perfect build' that will suit every player. But then, now here's the thing... if there is a perfect build out there that suits everyone... what's the point of having systems where you can make anything else? If you're not using the 'perfect build' then you are only ever hindering the people you are playing with and if you do that then people will call you out on it and shun you from content.
The problem with giving people these kinds of 'choices' in games isn't a mechanical one, it's not that it's an inherent problem within the system of the game but it is a sociological issue. It's the same reason why we have the 'crazy uncle' or 'crazy cat lady' archetypes. People who make a valid choice to live (or play) in a manner that is outside of what is deemed appropriate in a society will be ostracized by that society.
Yes, I don't want to have to deal with specialized builds, because ultimately it boils down to either 'you're doing it right' or 'you're doing it wrong'. Doing it one way will get you into content, doing it the other will keep you out of content. You don't even need to look very far to see this at a base level where you get the 'this composition vs that composition' argument over raids. When I play I want to be able to settle into my class and know that, beyond my gear setup, I don't have to sit and do the research or rely that someone out there is going to detail out every little thing I need to do, and I don't want to have to fix those skills when they make adjustments that change that build like every other patch if not more often.
Does it help to follow guides? Yes, looking up the things you will need as far as what gear you should have, what food you should bring and the like is one thing. But when you start putting in the variation of those skill choices it changes and suddenly it's, 'This build for this fight, this other build for the next one so go change your trees, then for the next one...' I had to do this once already in LotRO and it was hell, and I personally haven't stepped back into that game in the 2 years since I left it over that very issue.
I, for one, do not want to see XIV have to go this route, there are far better options available and I personally think that the Jobs are distinct enough as-is. I also don't want to have to worry even more about being in a DF group and wondering about what skills other players may or may not have. It's bad enough with Cross-Class skills going into a fight with a Tank-Swap mechanic and praying that the other tank knows enough to have Cross-classed Provoke on WAR/DRK if they've even taken the time to level GLD/PLD at all (yes, I have run into this problem on more than one occasion). If you're fine with this, and this is something that the devs choose to implement... which I highly doubt they ever would... then you can be free to enjoy it as much as you like, but for me in my opinion... as much as I love this game, I'll be moving on.