How about no? If you absolutely do not want to stay, eat the penalty or ask them to kick you (Most people will gladly oblige). No reason for the whole party to disband just because some people are impatient.