Yep I hate when that happens to me too. Even when I switch classes. Sometimes while fighting Ifrit.