Sadly, there's no cure for idiocy. I don't think there's a penalty in the game which will deter everyone who wants to quit at the start of a dungeon, and equally no reward which will encourage people to stick around if they were planning to bail.