As others have said, function-wise you're not really losing anything by going for a small house - you can still have:
- a fully functioning FC-workshop with the maximum amount of airships and submarines
- the option to buy private chambers
- a chocobo stable in the garden, aswell as other "useful" furniture just as a mailbox, an aerthyte, a training-dummy and a garden-bed

You're losing out a bit on the number of gardens you can have (only one for a smalle house, but if it only needs to sustain three people, you should be fine!) aswell as on "useful" indoor-items (there is a limit on the amount of crafting-stations you can place, they're giving your crafters a buff and are a little bit helpful during leveling but thats about it. They've increased the number of NPCs you can place in a small house recently, so you should be fine with getting all the important ones in! Note that the Butler/Maid/Hokonin sells some cheap furniture! Very, very helpful while decorating)

So a bigger house pretty much only gives you more space to decorate - but if you're already struggeling gil-wise, you are most likely fine with filling up a smaller space first.
Personally I've also made the experience that its easier to change a small house around - so I'd say thats more ideal for a small FC where everyone would like to get a shot at decorating at somepoint. I recently decorated my own mansion and I can tell you: I'm not gonna change that anytime soon since it would be an aweful lot of effort to remodel just one floor.
With the old small FC-house we could easly change around one floor in an afternoon to give it a new vibe, allowing more people to try their hands on that content! Now my FC has a mansion aswell and its kinda tricky to allow for everyone to decorate and still have it all fit together...