There's a couple of ways to swap inside the wards.

Changing wards in each district:
Mist: Storm Private at Mist South (Docks), or exit via Mistgate Square and select a new ward.
Lavender Beds: Serpent Private/Ferry Skipper at Amethyst Shallows
Goblet: Flame Private/Exit at Goblet North

Subdivisions (Plots 31-60):
If the ward you chose has the house you want taken, try the subdivision (plots 31-60) for that ward and see if that's available. Be aware that you can only get to the subdivision from the main ward's aetheryte crystals and unless something changes with the patch, even if you select the subdivision from the ward listing, you still get loaded into the main ward. EX: You see a house open in Ward 7 subdivision and select to go there, it will load you into Ward 7 and then you will be required to use the aetheryte crystals to load into the subdivision so you can see plots 31 to 60.