Just leave and find another one. There are many out there.

I was in my first FC and it was a good one - friendly, house, airship, etc - but the core group were often on at different times of day than I was. The rest of the FC seemed to be happy running around more of less solo. So through no fault of theirs I often felt like an outsider.
I can't imagine being in one where I avoided players in the FC. At that point why bother?

I left to join a very small FC (no house, airship, etc) and don't regret it at all.