Honestly, not being in an FC feels incredibly lonely in FFXIV, but if you want the quiet, you do you, man.

Anyway, I will note that if the invites are that gross and frequent, tagging yourself as busy is the only reliable way to halt invites completely. However, if you still need to be open to your friends, you can opt to either;
1) Make a linkshell for you and your friends.
2) Use the moogle mail system to communicate as needed.

Otherwise, there is nothing else for it.