Personally, it's all a part of playing a multiplayer game. Just speak up if the group is moving too fast, but be polite about it. If they agree to slow down, cool. If not, either put up with it or leave and find another group after the penalty runs out.

If you absolutely don't want to ever run with people who want to run at a different pace than you, use the Party Finder. It may take longer to get into the dungeon, but that's the price you pay to have it your way.

Having preferences that don't match with the preferences of others is one of the oldest causes of conflict in human history, this game is unfortunately not immune to that.