You could say it's just as much abuse to have people refuse to roll and then AFK. The worst instance was a player that stopped participating from the 2nd boss of ST2 onwards. Couldn't boot him because the alliance didn't want to wait and we'd be at the next boss before the timer expired (they were pretty good). He was obviously there, because as soon as Xande died, he lotted immediately, won the roll, and then exited the dungeon cuz he's an awesome person.
Abuse of the lotting system is no way to handle the very, very limited instances of vote kick abuse (i've actually never seen anyone kicked that didn't have good reason to get kicked)
I've also had it happen enough at the end of dungeons or primals to be annoying. Not game breaking, but annoying and would appreciate the fix.