You can actually always leave stuff with no penalty as long as the party isn't full(4,8), or the final boss is dead.