The right time to leave an FC is the moment you want to leave. You're not married to your FC so you can leave at a moments notice.

I've tried at least 20 different FCs and it just never clicked with me. Sometimes FCs aren't for people. I went solo for about a year before finally making my own FC simply cause I got tired of the random Blind Invites.

Also having an entire FC chest for yourself is amazing.