What do you mean by “has no goals”? Achieving an optimization target is goal-directed behavior.
This idea is interesting (to me) precisely because the goals of the algorithm don’t “care” about our universe, but rather about abstract mathematical questions, and the iteration is done in a way that may prevent “caring about our universe” from emerging as an instrumental subgoal. That’s a feature, not a bug.
But it wont do something else for the purpose of achieving an optimization target any more than an existing compiler would. See Blue Minimizing Robot:
Imagine a robot with a turret-mounted camera and laser. Each moment, it is programmed to move forward a certain distance and perform a sweep with its camera. As it sweeps, the robot continuously analyzes the average RGB value of the pixels in the camera image; if the blue component passes a certain threshold, the robot stops, fires its laser at the part of the world corresponding to the blue area in the camera image, and then continues on its way.
[...]
This is not because its utility function doesn’t exactly correspond to blue-minimization: even if we try to assign it a ponderous function like “minimize the color represented as blue within your current visual system, except in the case of holograms” it will be a case of overfitting a curve. The robot is not maximizing or minimizing anything. It does exactly what it says in its program: find something that appears blue and shoot it with a laser.
I still don’t understand your comment. Are you saying that the Oracle AI concept in general shouldn’t be thought of as AI? Or is it something with this particular proposal?
If the end result is a program that can output the source code of a Friendly AI (or solve other problems that we can’t currently write a program to solve), then what does it matter whether it’s an “AI” or an “agent” or not by some philosophical definition? (Similarly, if a program ends up forcibly rewriting the biosphere in order to count paperclips, it’s a moot point to argue over whether it counts as “intelligent” or not.)
Because it is safe to ask a non-agent Oracle questions which do not have dangerous answers (or create dangerous information while calculating the answer). On the other hand, an Oracle that behaves as an agent is unsafe to ask any question because it might convert the planet to computronium to calculate the answer.
OK, so you’re using “behaves as an agent” to mean what I mean by “cares about our universe”. It doesn’t sound like we disagree on substance (i.e. what such a program would do if it worked as proposed).
I’m not sure a system like this should be considered an AI as it has no goals and does not behave like an agent.
Do we distinguish between oracle AI and expert system?
edit: My lack of confidence is honest, not rhetorical.
What do you mean by “has no goals”? Achieving an optimization target is goal-directed behavior.
This idea is interesting (to me) precisely because the goals of the algorithm don’t “care” about our universe, but rather about abstract mathematical questions, and the iteration is done in a way that may prevent “caring about our universe” from emerging as an instrumental subgoal. That’s a feature, not a bug.
But it wont do something else for the purpose of achieving an optimization target any more than an existing compiler would. See Blue Minimizing Robot:
[...]
I still don’t understand your comment. Are you saying that the Oracle AI concept in general shouldn’t be thought of as AI? Or is it something with this particular proposal?
If the end result is a program that can output the source code of a Friendly AI (or solve other problems that we can’t currently write a program to solve), then what does it matter whether it’s an “AI” or an “agent” or not by some philosophical definition? (Similarly, if a program ends up forcibly rewriting the biosphere in order to count paperclips, it’s a moot point to argue over whether it counts as “intelligent” or not.)
Because it is safe to ask a non-agent Oracle questions which do not have dangerous answers (or create dangerous information while calculating the answer). On the other hand, an Oracle that behaves as an agent is unsafe to ask any question because it might convert the planet to computronium to calculate the answer.
OK, so you’re using “behaves as an agent” to mean what I mean by “cares about our universe”. It doesn’t sound like we disagree on substance (i.e. what such a program would do if it worked as proposed).