Yeah, it definitely is annoying. I did it accidentally yesterday in a dungeon, which sucked. It doesn't happen often, but is noticeable when it does.