That's seriously your complaint?
Here's a fix.
When someone issues you a duel no matter wait, it only appears as the small minimized confirmation window. Like the little box that appears when you hit wait for the duty finder.
You just assume that if someone issues you a duel you're going to be unable to take any actions until hitting accept or decline? That's kind of silly don't you think. That's not how it would be and you know it.