Diversion and Lucid Dreaming are very different. Diversion reduces the amount of threat your skills generate, while Lucid Dreaming halves your current amount of threat. One is proactive and used BEFORE you pull aggro so you give the tank time to lock threat on mobs. The other is reactive and used AFTER you've already pulled aggro. Lucid Dreaming is your "I screwed up" button as far as threat is concerned IMO.
Personally I take BOTH skills. Diversion is my opener in any situation where I feel its likely I'll pull aggro and make it difficult for the tank to round things up and lock them down. Lucid Dreaming is something I shouldn't have to use unless I need mp but if aggro does get dicey, I have it to drop my threat in an instant. I feel that if I didn't have Diversion and simply relied on Lucid, I'd be slowing down the group overall because mobs would be breaking off from the tank, messing up positionals for the melee, forcing the tank to wrangle them back and overall just hurting the group's efficiency. Prior to the role skill revamp, caster dps had Diversion as their only threat management tool and I feel its the primary tool caster dps are intended to use for threat management, while Lucid is meant more for mp management and a bonus threat dump for emergencies.
Diversion is far from a wasted job action and it's certainly better than most other skills you could take in its place, especially if you're actually having threat issues. If you're having aggro problems but don't take you aggro management tool because you think its "a wasted job action" then the problem is you. Now that's not to say that some tanks aren't bad at their jobs or spending too much time in dps stance, but if you're consistently having threat issues no matter the tank, the issue is probably not the tanks.