If the party is missing a member, regardless of why, you won't take a penalty. That said, if the spot fills in the seconds between you choosing to leave and actually being removed, then you're considered a deserter.