I relocated my FC house to Ishgard. I want to replace the old Estate Hall with a new one. I bought the permit.

When I click "Build Estate Hall" it says "Unable to build. The current estate hall must first be removed."

When I click "Remove Estate Hall" it says "Unable to remove estate hall before all private chambers have been demolished."

Am I supposed to demolish the private chambers of all my FC members to change the Estate Hall?

How?