You can do exactly that. By default, they only have the right to warp to your house and use your Npcs (Crystal bells, material suppliers, armoire.)

You can expand those rights to gardening and chocobo Stables. Just dont enable the remove and place furniture option. :]