Not all men enjoy walking around with a bare chest. Shocking! I know!
Besides, while I can't speak for gay men; I know that not all women care to see a bare chested men either. Because as you say its seen as sex appeal. For that same reason many men wouldnt care to see it either.
But I'll speak for gay men anyway, and claim that not every gay man would want to see it either, because of sex appeal.
Would you want to see a bare chested woman in ffxiv for example? I mean I don't play the game to see literally half naked characters.
If people want to dress up as such that's up to them, but please dont suggest changes to an item to bring out its sex appeal.