I've left a FC before and a simple "It's not the right fit for me" was enough to not leave anyone terribly upset. I still have friends from there.

We have people join and then leave our FC somewhat often. It's no big deal. I just assume we weren't the right fit for their play style or personality.

Search around, find your people.