you normally don't incur a penalty for leaving other 4 man content... EXCEPT for Roulettes. The penalty is put there so that you can't just easily leave and fish for something that you want to do over something you don't want to do. This is how it's always been. You should have waited for the vote kick 5 min timer instead.