I figured out a way around it:
Go to the nearest exclamation mark, click on the npc and get the accept/decline box to pop up, then go afk and you won't get booted.