People are allowed to kick you out of their own parties (hunts/shinryu/savage groups). So the only thing you have a leg to stand on for is dungeon pugs, and even then, difference in playstyle is a thing.