Simple fix is to not show you are getting into a party in progress. If people want to withdrawing they will get the 15 min burn.