I think the majority of male players that play female characters have no real issue with females playing as males. I hope you'll understand that your experiences are likely due to not encountering situations where people don't speak up about it.