I feel like capital markets are drooling over the potential to permanently eliminate yet another labor cost. Even if an AI is doing the same thing at people are in terms of training, I'm not using an ethical framework that requires that I protect the right and livelihood of an AI. Humans have a societal duty to protect humankind and no such requirement for AI.

And to a prior point regarding tools, a nuclear bomb is a tool. It's an inherently destructive tool. Chemical weapons are a tool. An inherently destructive one. Some tools have proclivities, and we regulate them heavily because those proclivities are harmful.

AI also isn't like prior tools. Industrial earth movers replaced a lot of digging as a job. One tool changes how we do the job but it's still people doing the job, just at increased scale. AI threatens to replace the human entirely as the skill-learning entity eliminating the labor entirely replacing it with capital. Instead of someone you hire, it's something you own. My prediction is that AI is going to accelerate massively the movement of wealth from laborers to capital-owners beyond anything we've ever seen.