If you did the whole dungeon with only 4 people and no one else joined, it means one of you (the leader) did not click the recruit more party members button. I have never had to wait more than 1 or 2 minutes for the instance to repopulate, even if almost everyone else has left.