Unless as stated above, the party is full again. If you leave with a full group, even if someone had left the group prior, you get the penalty.