Superintelligence, the book that inspired Elon Musk's warnings about AI

I came across Superintelligence by Nick Bostrom because Elon Musk recommended it years ago when he was starting to talk a lot about the risks of AI. I knew roughly what the book was about, but I hadn't really looked into the actual arguments behind it. Some of them are pretty interesting, especially now that AI is moving as fast as it is. The basic question is: what happens if we build an AI that's smarter than us? And not just better at answering questions or writing code. Bostrom is talking about something better than humans at basically everything intellectual: science, engineering, strategy, persuasion, research, planning, etc. One idea that stuck with me is that AI doesn't have to become evil to be dangerous. It doesn't need to hate us. It could just be extremely good at doing something we asked it to do, while completely missing what we actually meant. There's the famous paperclip example. You tell a superintelligent AI to maximize paperclip production. It needs resources to make paperclips, so it builds more factories. Then it needs more energy and materials. Eventually almost everything becomes useful for making more paperclips. Humans might get in the way. Obviously nobody is going to build an AI whose only purpose is making paperclips. That's not really the point. The point is what happens when a very powerful system follows the literal objective you gave it instead of the intention behind it. Another part of the book is about an "intelligence explosion". If an AI becomes good enough at AI research, at some point it might be able to improve itself. The improved version would then be better at improving itself again. Nobody really knows how fast that process could happen, but Bostrom's argument is that human-level AI might not stay human-level for very long. That's where the control problem starts getting weird. If you have something much smarter than the people trying to control it, how exactly do you keep it under control? Even things like acquiring more resources, preventing itself from being switched off or improving its own capabilities could make sense from the AI's point of view because they help it achieve whatever objective it has. You don't necessarily have to program "survive" or "gain power" into it directly. The part I probably find most interesting is the problem of actually telling an AI what we want. It sounds easy until you try to define it. "Make humans happy." Okay, what exactly counts as happiness? "Keep humans safe." How safe? Would locking everyone in a perfectly controlled environment technically satisfy that? "Improve humanity." According to whose definition of improvement? Humans can't even agree among ourselves on a lot of these questions, yet with a superintelligent system we'd somehow have to translate them into objectives precise enough that the system can't optimize them in some completely unintended way. And that's basically why alignment matters so much in the book. Bostrom isn't saying superintelligence has to be bad either. If we actually managed to build one and align it properly, the upside would be ridiculous. Medicine, physics, energy, space, engineering... you could potentially speed up scientific progress by an insane amount. It's the combination of those two possibilities that makes the book interesting. Best case, it could be one of the greatest things humanity ever creates. Worst case, we create something more capable than ourselves before we've figured out how to control what it wants. The book came out in 2014. Reading the ideas now feels very different than I imagine it did back then.