That's a dangerous game, because if you just watch while three other people kill the adds -- and they survive without you -- you've just confirmed to three separate people that you aren't really needed.
That usually doesn't happen (the errant puller surviving, I mean; the vast majority of the time I see this happen when I'm tanking, they end up dying and immediately leaving the dungeon without a word), and quite frankly, as a tank I really don't care if a group decides to kick me. I will get another group far easier than the majority of them.

Not to mention it's a lot more appealing for a DPS player to kick the tank than to kick the healer, if it comes down to that.
The majority of melee DPS will do crappier DPS without someone to reliably hold hate for them so they can hit their positionals. Healer DPS will also be compromised too because they will have to do more babysitting.

It's really not "appealing" for DPS to kick their tank just because his pacing is a little slower than they'd like. If they're really just standing around and being useless most of the time, then sure, but otherwise they're usually just shooting themselves in the foot.