Direct question. I cannot infer answer from you posts. If human values do not exist in closed form (i.e. do include updates on future observations including observations which in fact aren’t possible in our universe), then is it better to have FAI operating on some closed form of values instead?
I don’t understand the question. Unpack closed form/no closed form, and where updating comes in. (I probably won’t be able to answer, since this deals with observations, which I don’t understand still.)
[...] computer [operating within Conway’s game of life universe] is given a goal of tiling universe with most common still life in it and universe is possibly infinite.
One way I can think of to describe closed/no closed distinction is that latter does require unknown amount of input to be able to compute final/unchanging ordering over (internal representations of) world-states, former doesn’t require input at all or requires predictable amount of input to do the same.
Another way to think about value with no closed form is that it gradually incorporates terms/algorithms acquired/constructed from environment.
I understand the grandparent comment now. Open/closed distinction can in principle be extracted from values, so that values of the original agent only specify what kind of program the agent should self-improve into, while that program is left to deal with any potential observations. (It’s not better to forget some component of values.)
I’m not sure I understand you. Values of the original agent specify a class of programs it can become. Which program of this class should deal with observations?
It’s not better to forget some component of values.
Forget? Is it about “too smart to optimize”? This meaning I didn’t intend.
When computer encounters borders of universe, it will have incentive to explore every possibility that it is not true border of universe such as: active deception by adversary, different rules of game’s “physics” for the rest of universe, possibility that its universe is simulated and so on. I don’t see why it is rational for it to ever stop checking those hypotheses and begin to optimize universe.
Direct question. I cannot infer answer from you posts. If human values do not exist in closed form (i.e. do include updates on future observations including observations which in fact aren’t possible in our universe), then is it better to have FAI operating on some closed form of values instead?
I don’t understand the question. Unpack closed form/no closed form, and where updating comes in. (I probably won’t be able to answer, since this deals with observations, which I don’t understand still.)
Then it seems better to demonstrate it on toy model as I’ve done for no closed form already.
One way I can think of to describe closed/no closed distinction is that latter does require unknown amount of input to be able to compute final/unchanging ordering over (internal representations of) world-states, former doesn’t require input at all or requires predictable amount of input to do the same.
Another way to think about value with no closed form is that it gradually incorporates terms/algorithms acquired/constructed from environment.
I understand the grandparent comment now. Open/closed distinction can in principle be extracted from values, so that values of the original agent only specify what kind of program the agent should self-improve into, while that program is left to deal with any potential observations. (It’s not better to forget some component of values.)
I’m not sure I understand you. Values of the original agent specify a class of programs it can become. Which program of this class should deal with observations?
Forget? Is it about “too smart to optimize”? This meaning I didn’t intend.
When computer encounters borders of universe, it will have incentive to explore every possibility that it is not true border of universe such as: active deception by adversary, different rules of game’s “physics” for the rest of universe, possibility that its universe is simulated and so on. I don’t see why it is rational for it to ever stop checking those hypotheses and begin to optimize universe.