If you just go with the party when the teleporter is open, you would sooner get to your next potd run and get more silver chests. But it seems you must have them now.

There is really no need to fully explore the whole area. You'll get more chances in the future.