There should be a "are you sure you want to invite <player> to your cross world linkshell?" prompt after you click that option on accident.
Its very easy to misclick that option when opening party interface and right clicking on the party finder host to read their recruitment ad.
https://i.imgur.com/nzk2RFR.png
https://i.imgur.com/sfjWZRL.png
phew thank god. That would've been awkward otherwise.