The right time to leave is when these two conditions are met:
1. you are starting to feel frustrated beyond a level of wanting to overcome a challenge
2. You have informed your party that you'll leave if the problems persist through the next one or two fights.

Don't just silently eff off in the middle of a run, that's rude to the innocent party members. Unless all three/seven of your party members are assholes, of course, in which case, make like a tree.