I share your exact concerns. What are the chances of having exactly 8 people show up for "dungeon-night"? Very low. You'll most likely have more or less. Quite an odd game design decision to be honest.
Also, if its as difficult as they claim, then picking up Random Joe will most likely be detrimental to the completion of the dungeon.
One more point, if the gear is as awesome and rare as SE claims then most of the time LSs will want some of loot system (formal or informal). Even more reason to not invite Random Joe.
There are so many reasons to NOT have a minimum party size, yet none for having a minimum req.(well, peregines concern about elitism, I'll agree to disagree on that point).