I would think the sweet spot for some harder content would be like 12-16 players, it will allow for slightly more variation in team compositions and mechanic design, while not going all the way to the alliance/eureka/bozja level of mess. Sadly the party UI isn't really designed for that.

4 players is great to reduce the bar of getting in, but it will most likely end up having to deal with simpler team mechanics and having most of the responsibility fall onto the single healer. DPS are balanced around the 8 people content anyway so some of the might struggle reaching the DPS checks which basically means those will either be super lenient or so hard that you have to bring spesific Jobs. Tanking will be completely brain dead unless there's insane amount of enemy positioning through the fights because there's no swaps among other stuff for 4man.