Mounts are effectively magic anyway, unless you assume that we all have some chocobos, a few horses, some lizards, some mecha, a car, an airship, a bird or two, and the king of dragons just kinda lurking off-screen waiting for us to whistle. Maybe that cast time is for all of them to figure out which one you're trying to call.
Just be glad it's not like early WoW where you had to keep the whistle in your bag to use it.