You have to remove everything. This includes unique items that get destroyed on pick-up. This includes the chocobo stable (which can't be removed unless every chocobo is removed). Every personal chamber must be demolished, which means that every item from every personal chamber must be removed. You can kick people out of the FC to solve the chocobo and chambers issues. You also lose all progress on airships/workshop. And you will need to buy the new plot and house for the full amount. You get no gil back from relinquishing the current plot.

Now, there is a Move feature coming out in a future patch, though we don't know what patch it will be or which (if any) of these problems it will solve. Depending on how eager your FC is to move (and how many plots you would be interested in are still available), you might want to hold out for the Move feature.