You're not in the wrong. Every online community will have rude people. You just need to ignore them and play the game your own way.

If it gets too much try take a break from dungeons and try something else. And if they go over the top don't be afraid to report them.