Whether they were right or not, if you kept doing something they asked you not to do, that (they believed) was causing enemies to attack the party, then from their perspective it would seem like trolling.

And if you need to walk slowly past enemies to avoid them, then it does seem logical that a "move faster" spell could be the difference between passing safely and drawing their attention. Particularly with the "evidence" that they do get attacked.

The information might be incorrect, but it doesn't sound entirely unreasonable in those circumstances.