I don't believe it's possible to relinquish the estate until all chocobos are removed from the stables and personal rooms are removed (inactive members would have to be kicked). The move feature gets around this, but if you're house swapping with another fc one of you is going to have to start culling members that aren't on.