Seems it happens most times when I enter a very populated city. Probably an error due to loading characters? It's happening constantly on Limsa Lominsa but out in the field it never happens.