You need to decide if this is true:
Or if this is true:
Because, literally, they are mutually exclusive. Either a Ninja really is 'doing it for you' and you have had "the most fun [you've] had in this game", or it isn't. You need to be honest with yourself and forget what your friends or siblings might say, forget what your favorite streamer might play, forget about the article or Reddit thread you read that said Ninjas are the best thing since sliced bread, and just actually do what you want to do.
If the 'most fun you've had in the game' is a Ninja, as you stated, then the armor and fantasy is sufficient for you to have a really good time. Hence, you'd continue to play a Ninja.
If the armor and fantasy are bringing you down, then it can't possibly be 'the most fun you've had in the game.' Well, I mean, it could be, but if the most fun you've had is playing a character that 'doesn't do it for you,' something is amiss. IMHO you'd drop the Ninja, and try a different job. Possibly a different game if that less-than-stellar experience was the best fun you've had here.
Same as if someone insisted you eat a plate of oysters, and you didn't like them after trying them, you'd say, "no, thank you," instead of forcing them down. Well, at least I would.
Personally, I don't do entertainment I don't like. Music, movies, books, games ... if I don't like those things I don't continue to partake of them, and I don't do the 'everyone else likes this so I should too,' thing. Done and dusted with that about 30 years ago.
Either you like Ninja enough to play it, or you don't. So, you either play it or not based on your internal feelings. To be honest, my whole post here is irrelevant relative to your direct questions. You shouldn't either play or not play a Ninja based on what other people would do or have done. You do you.