What is point of having them to be need in party?
I do fates people go around do stuff and we complete it without being in party.

Only logical reason for you needing to be in party is as stated before :"I can't do the bare minimum, let me level afk without getting kicked plz!"