What you're describing I would actually consider harrassment and very childish behaviour - however if the majority wants to progress and not look for chests and expresses that desire in a civilised manner ("Please come to the lamp, we would like to go on") it isnt harrassment since no on in PotD has to go out of their way to get chests for you. By running off alone why three other people are waiting at the teleporter, you're activley holding those three people back, thats not okay either.
The simple conflict of intrest that might present itself in PotD doesnt make one side the bad guys and the other one the holy saints - no one is "right" or "wrong" in their desire how to tackle this content. How that desire is expressed is relevant however. Coming to a mutual agreement is relevant - and yes, that agreement can be "The majority would like to do it this way, please follow".
People throwing a tantrum - on either side, wether they want chests or not - are in the wrong and if someone actually resorts to harrassing and griefing you should report them.
But, again: Simply saying "No, three of us dont want to go after silver chests, so we'll be heading straight for the next floor" is neither harrassment nor griefing.
And why is it that someone who doesnt want to explore the whole floor has to go out of their way?
Person A&C want chests. Person B&D wants exp (and would therefor like to proceed faster). By your logic everyone has to go after chests now (they even have to go after chests if only Person A wants them), but why?
My personal proposal would always be a compromisse - maybe something like "We will be looking for chests on the first 5 floors and skip right to the lamp on the last 5" - how does that sound to you?
The real solution here is always to find a way that works for the majority of the group and benefits the most of the present people.
If you have one person (from either side btw!) that is extremly uncooperative and resorts to harrassing and active griefing: remove and report them.
Try to reach an agreement with your group - but be willing to go out of your own way aswell. Dont go into DF and expect everyone to obey to your rules and goals.