I have had this happen very often. Channel any skill with a long cast while you're moving with swiftcast, and every once in a while, one will just get interrupted somehow, and waste the buff.