I’m not entirely sure what your idea of oracle is supposed to do, though. Metaphorically speaking—provide me with a tea recipe if I ask “how to make tea”?
Bingo. Without doing anything else other than answering your question.
So, for the given string Q you need to output a string A so that some answer fitness function f(Q,A) is maximized. I don’t see why it has to involve some tea-seeking utility function over expected futures. Granted, we don’t know what a good f looks like, but we don’t know how to define tea as a function over the gluons and quarks either. edit: and at least we could learn a lot of properties of f from snooped conversations between humans.
Yes, that model is a good model. There would be some notion of “answer fitness for the question”, which the agent learns from and tries to maximize. This would be basically a reinforcement learner with text-only output. “Wireheading” would be a form of overfitting, and the question would then be reduced to: can a not-so-super intelligence still win the AI Box Game even while giving its creepy mind-control signals in the form of tea recipes?
Bingo. Without doing anything else other than answering your question.
I think the important criterion is lack of extensive optimization of what it says for the sake of creation of tea or other real world goal. The reason I can’t really worry about all that is that I don’t think a “lack of extensive search” is hard to ensure in actual engineered solutions (built on limited hardware), even if it is very unwieldy to express in simple formalisms that specify an iteration over all possible answers. The optimization to make the general principle work on limited hardware requires to cull the search.
There’s no formalization of Siri that’s substantially simpler than the actual implementation, either. I don’t think ease of making a simple formal model at all corresponds with likelihood of actual construction, especially when formal models do grossly bruteforce things (making their actual implementation require a lot of effort and be predicated on precisely the ability to formalize restricted solutions and restricted ontologies).
If we can allow non-natural language communication: you can express goals such as “find a cure for cancer” as a functions over fixed, limited model of the world, and apply said actions inside the model (where you can watch how it works).
Let’s suppose that in the step 1 we learn a model of the world, say, in Solomonoff Induction—ish way. In practice with the controls over what sort of precision we need and where, because our computer’s computational power is usually a microscopic fraction of what it’s trying to predict. In the step 2, we find an input to the model that puts the model into desired state. We don’t have a real world manipulator linked up to the model, and we don’t update the model. Instead we have a visualizer (which can be set up even in an opaque model by requiring it to learn to predict a view from arbitrarily moveable camera).
Bingo. Without doing anything else other than answering your question.
Yes, that model is a good model. There would be some notion of “answer fitness for the question”, which the agent learns from and tries to maximize. This would be basically a reinforcement learner with text-only output. “Wireheading” would be a form of overfitting, and the question would then be reduced to: can a not-so-super intelligence still win the AI Box Game even while giving its creepy mind-control signals in the form of tea recipes?
I think the important criterion is lack of extensive optimization of what it says for the sake of creation of tea or other real world goal. The reason I can’t really worry about all that is that I don’t think a “lack of extensive search” is hard to ensure in actual engineered solutions (built on limited hardware), even if it is very unwieldy to express in simple formalisms that specify an iteration over all possible answers. The optimization to make the general principle work on limited hardware requires to cull the search.
There’s no formalization of Siri that’s substantially simpler than the actual implementation, either. I don’t think ease of making a simple formal model at all corresponds with likelihood of actual construction, especially when formal models do grossly bruteforce things (making their actual implementation require a lot of effort and be predicated on precisely the ability to formalize restricted solutions and restricted ontologies).
If we can allow non-natural language communication: you can express goals such as “find a cure for cancer” as a functions over fixed, limited model of the world, and apply said actions inside the model (where you can watch how it works).
Let’s suppose that in the step 1 we learn a model of the world, say, in Solomonoff Induction—ish way. In practice with the controls over what sort of precision we need and where, because our computer’s computational power is usually a microscopic fraction of what it’s trying to predict. In the step 2, we find an input to the model that puts the model into desired state. We don’t have a real world manipulator linked up to the model, and we don’t update the model. Instead we have a visualizer (which can be set up even in an opaque model by requiring it to learn to predict a view from arbitrarily moveable camera).