Duty Finder Window Popup Ps

This is quite aggravating and I'm sure I'm not the only victim of this but when duty finder pops up to commence the duty, the window and interactions are taken from whatever you are doing at that moment to the duty window. I just this morning lost my place in a duty because I was depositing things to my retainer and it flashed up but I was pressing right and x to deposit so fast I withdrew myself in the blink of an eye.

This is really aggravating, why does it need to do this? It's fine if your standing around not doing anything but if your in a merchant window etc it swaps your control from one to another often before I even notice.