But it can’t, by definition, invent a new tool entirely.
Why not? The AI can do anything that is allowed by the laws of physics, and maybe a bit more if we let it. It could invent a molecule which acts as a drug which kills the cancer. It could use the tools in the room to build different tools. It could give us plans for tiny nanobots which enter the bloodstream and target cancer cells. Etc.
Just because an environment is well defined, does not mean you can’t invent anything new.
The AI can do anything that is allowed by the laws of physics
No, it can do anything that its worldview includes, and any operations defined internally. You’re no longer talking about a Godel Machine, and you’ve lost all your safety constraints.
You can give, in theory of course, a formal description of the laws of physics. Then you can ask it to produce a plan or machine which fulfills any constraints you ask. You don’t need to worry about it escaping from the box. Now it’s solution might be terrible without tons of constraints, but it’s at least not optimized to escape from the box or to trick you.
Why not? The AI can do anything that is allowed by the laws of physics, and maybe a bit more if we let it. It could invent a molecule which acts as a drug which kills the cancer. It could use the tools in the room to build different tools. It could give us plans for tiny nanobots which enter the bloodstream and target cancer cells. Etc.
Just because an environment is well defined, does not mean you can’t invent anything new.
No, it can do anything that its worldview includes, and any operations defined internally. You’re no longer talking about a Godel Machine, and you’ve lost all your safety constraints.
You can give, in theory of course, a formal description of the laws of physics. Then you can ask it to produce a plan or machine which fulfills any constraints you ask. You don’t need to worry about it escaping from the box. Now it’s solution might be terrible without tons of constraints, but it’s at least not optimized to escape from the box or to trick you.