You get a second of "1", then half a second of " " before it comes off. Watch your own video, you were at least halfway through the "1", leaving you less than a second. Like I said, try it out with movement (and moving during swiftcast doesn't count).
In that scenario you're talking ~70 Piety, which is -70 crit best case. That's more than 1% less damage per cast, way more than your rotation can hope to gain. And that's before factoring in a B1 cast.
