You're right to be angry. This has been a long going problem, which is the reason why they changed how the final dungeons happen in Heavensward and Stormblood. Unfortunately, this is the norm for people now. People will actually kick others if they want to watch the CS. It's disgusting, and I really wish they'd do something like split the dungeon into a bunch of trials or something.