4 people should not be optimal, otherwise people would exclude others from their party. There should always be an incentive to bring in more people who need xp into the party.

I'd suggest having the XP/hour reach a certain level once you have 4 people in your party, then, if a friend of any other people want to join they don't hurt your xp/hour.