There's an area in the north of Gridania that we're not allowed to enter and the guard explicitly says it's residential and outsiders are not allowed in. There's also an insane amount of buildings in Ul'Dah we can't go into, either, presumably houses. Although Ul'dah itself is the most justified since it (along with the Brume in Ishgard) has a sizeable beggar/homeless situation.

Conservation of detail and resources, mostly