I had this problem too, with 1, 2, and 4 people.

If it wasn't just a "Cannot enter at this time" message, it said "the hands of darkness prevent you from entering".