I'm 99.99999% sure it defaults to "Accept", so if you are spamming X you get queued.

The only way to cancel it is if you press left or right and then X very quickly (like, right as the sound plays for the queue popping). Which then IDK what to tell you but maybe slow down? Even hitting O doesn't cancel it right away (just minimizes the window)