Phantom Jobs are a step in the right direction but need iteration.

[Yamadronis]

I remember someone unintentionally grief the party by granting swift buff when people are trying to dodge aoe, overshot, and run into the death wall instead XD

As the kids are wont to say, "Skill Issue". But no yeah if your movement speed suddenly increases without warning it's absolutely understandable that it might kill you. This is probably why the speed boosts are limited to self and single target in the phantom jobs.