Maybe the AI can make its first nanomachines easy to reason about… but maybe not. We humans cannot predict the outcome of even relatively simple chemical interactions (resorting to the lab to see what happens). That’s because these chemical interactions are governed by the laws of quantum mechanics, and yes, they involve superpositions of a large number of states.
“Its possible the AI shows P!=NP, but also possible the AI shows P=NP, and finds a fast algorithm. Maybe the AI realizes that BQP=BPP.”
It’s also possible the AI finds a way to break the second law of thermodynamics and to travel faster than light, if we’re just gonna make things up. (I have more confidence in P!=NP than in just about any phsyical law.) If we only have to fear an AI in a world where P=NP, then I’m personally not afraid.
Not sure why you are quite so confident P!=NP. But that doesn’t really matter.
Consider bond strength. Lets say the energy taken to break a C-C bond varies by ±5% based on all sorts of complicated considerations involving the surrounding chemical structure. An AI designing a nanomachine can just apply 10% more energy than needed.
A quantum computer doesn’t just have a superposition of many states, its a superposition carefully chosen such that all the pieces destructively and constructively interfere in exactly the right way. Not that the AI needs exact predictions anyway.
Also, the AI can cheat. As well as fundamental physics, it has access to a huge dataset of experiments conducted by humans. It doesn’t need to deduce everything from QED, it can deduce things from random chemistry papers too.
Maybe the AI can make its first nanomachines easy to reason about… but maybe not. We humans cannot predict the outcome of even relatively simple chemical interactions (resorting to the lab to see what happens). That’s because these chemical interactions are governed by the laws of quantum mechanics, and yes, they involve superpositions of a large number of states.
“Its possible the AI shows P!=NP, but also possible the AI shows P=NP, and finds a fast algorithm. Maybe the AI realizes that BQP=BPP.”
It’s also possible the AI finds a way to break the second law of thermodynamics and to travel faster than light, if we’re just gonna make things up. (I have more confidence in P!=NP than in just about any phsyical law.) If we only have to fear an AI in a world where P=NP, then I’m personally not afraid.
Not sure why you are quite so confident P!=NP. But that doesn’t really matter.
Consider bond strength. Lets say the energy taken to break a C-C bond varies by ±5% based on all sorts of complicated considerations involving the surrounding chemical structure. An AI designing a nanomachine can just apply 10% more energy than needed.
A quantum computer doesn’t just have a superposition of many states, its a superposition carefully chosen such that all the pieces destructively and constructively interfere in exactly the right way. Not that the AI needs exact predictions anyway.
Also, the AI can cheat. As well as fundamental physics, it has access to a huge dataset of experiments conducted by humans. It doesn’t need to deduce everything from QED, it can deduce things from random chemistry papers too.