I just got done with Haukke Manor, and had two tanks bail right after we got the first two keys (and two chests) they each basically came in to grab loot, then left our party. I say no, do not take away penalty, in fact it should be more harsh.