They don't need to have access to gardening
Actually, most players still wants to do gardening, considering it's a housing benefits, which is something that is shared by FC members. There's a LOT of people that's wanting to do garden but couldn't, but that's another topic.

What you're suggesting is very similar to how private rooms work anyway. :/