I found one of those so far. Just ignore him and if he keeps going, go afk inside the dungeon.