The withdrawal counts towards everyone in the party, not just the person that withdrew. Had it happen to my group today running dungeons to lvl rog/nin. People in the party had to withdraw for various reasons and when we hit the third time we all got penalized with a 30 min lockout even though it was different people that did the withdrawing.