Let's assume we use our friend list to assign privilege like how you currently do for FC house. Since there is no rank system in friend list (e.g. best friend, hi-bye friend). You can only assign the particular privilege to all or none of your friend.

Privileges1: Using stable = OK
Privileges2: Plant and harvest garden = Dangerous
Privileges3: Decorating house, removing furniture, change exterior = Dangerous
Privileges4: Setting access rights to my house = Dangerous

Most of us have large friend list that has people we called friend but not enough to trust our entire house to them. I don't hand my house key to my friend in r/l either.

So to make this work in the current system, I will have to remove most of my friends.

Or SE needs to setup some kind of ranking system that allows me to categories all my friends and assign them different privileges so we could share our resources. But I think that's called setting up a Free Company.