I do foresee one potential way the votes could be 'rigged' and that's through using the party in progress queue... Have a 4-man pre-made team enter, have one leave, then have them use the in-progress queue to get back in. Clear the dungeon and everyone votes for the one that joined after.

It'd be a pain the backside to do and probably wouldn't work reliably in the busier dungeons but just thought I'd point out that it would be possible (unless they disable it for the in-progress queue).