This also happens to me! It seems to only happen in dungeons or areas heavily populated. Never on the world maps.

It makes no sense, no other game does this to me.