Also the person you are inviting cannot be in another party at the time. And they will not be able to accept if they are "busy" - talking to an NPC, watching a cutscene, crafting,..