Seems like the new guy accidentally put you into leveling roulette instead of Haukke Manor, which is why you got the penalty. Vote abandon is always the safer option, though you understandably didn't think there was any risk in just leaving.