The way I see it, if the group is sitting on the teleport and you are trying to explore for more chests, they are well wthin thier rights to ask you not to. And if you refuse, they are well within thier rights to remove you from the party for difference in playstyle.

Go with the majority, or expect to be singled out.