It's probably the same reason you can't accept a party invite while engaged with your retainer (or other NPCs):

Spaghetti code.