Downside is you'll probably wind up casting spells that depend on some kind of potency in fast cast gear. If it's not something constant or controlled, no thanks.