You don't lose target when someone casts invisible, just when they swap gear. Your auto-follow cancels, however.

But that's what target-lock is for :3