Strong world-optimization only happens if there is a robust and strong correlation between the world-model and reality.
Humans and corporations do not have perfect world models. Our knowledge of the world and therefore our world models are very limited. Still humans and corporations manage to optimize. Mostly this happens by trial and error (and copying succesful behaviors of others).
So I wonder if strong world-optimization could occur as an interative process based on an imperfect model of the world. This however assumes interaction with the world and not a “just in your head” process.
As a thought experiment I propose a corporation evading tax law. Over time corporations always manage to minimize the amount of taxes paid. But I think this is not based on a perfect world model. It is an iterative process whereby people predict, try things and learn along the way. (another example could be the scientific method, also iterative and not in your head but there is an interaction with the world).
My claim however assumes that optimization not occuring just in your head, but interaction with the real world is neccessary for optimization. So maybe I am missing the point of your argument here.
I didn’t understand the technical details of the article, but this seems correct.
If you have a perfect model with zero uncertainty, you can solve the entire situation in your head, and then when you actually do it, the result should be the same… or the assumptions were wrong somehow.
Otherwise, I think it makes sense to distinguish two types of situations:
a) During the execution of the plan, something completely unexpected happens. Oops, you have to update, and start thinking again, considering the new information.
b) Your model has some uncertainty, but you know the statistical distributions. For example, with probability 80% the world is in state X, with probability 20% it is in state Y, but you cannot immediately check which option it is. But you can, for example, create a five-step plan based on the (more likely) assumption that it was state X, and if the assumption is wrong, you know it will become visible during step 3, in which case you will switch to alternative steps 4b and 5b. Or if the switch would be too expensive, maybe you could instead add a step zero, some experiment which will figure out whether it is X or Y.
The difference is between “I didn’t expect this—must update and think again” and “I expected this could happen—switching to (already prepared) Plan B”. The former requires iteration, but the latter does not.
An analogy in computer programming would be a) the programmer finding out that the program has a bug and trying to fix it; vs b) the programmer including an “if” statement or exception handling in the program.
In real life the distinction can be less clean. For example, even if you have exact statistical distributions, the resulting number of combinations may be too large to handle computationally, so you might prepare plans for the three (or thirty, if you are a superintelligence) most likely scenarios in advance, and stop and think again if something else happens.
On the other hand, even when unexpected things happen, we often do not immediately freeze and start thinking, but try to stabilize things first. (If I suddenly find during my walk that I am walking in a wrong direction, I will first put both my feet on the ground; and maybe if I am in the middle of a road, I will get to the sidewalk first… and only then I will look at the map and start thinking.) This again assumes that I have correct probability distributions at least about some things (that putting both feet on the ground will increase my stability; that standing on the sidewalk is safer than standing in the middle of a road) even if I got something else wrong (the street I am currently on).
Your model has some uncertainty, but you know the statistical distributions. For example, with probability 80% the world is in state X, with probability 20% it is in state Y.
Humans and corporations do not have perfect world models. Our knowledge of the world and therefore our world models are very limited. Still humans and corporations manage to optimize. Mostly this happens by trial and error (and copying succesful behaviors of others).
So I wonder if strong world-optimization could occur as an interative process based on an imperfect model of the world. This however assumes interaction with the world and not a “just in your head” process.
As a thought experiment I propose a corporation evading tax law. Over time corporations always manage to minimize the amount of taxes paid. But I think this is not based on a perfect world model. It is an iterative process whereby people predict, try things and learn along the way. (another example could be the scientific method, also iterative and not in your head but there is an interaction with the world).
My claim however assumes that optimization not occuring just in your head, but interaction with the real world is neccessary for optimization. So maybe I am missing the point of your argument here.
I didn’t understand the technical details of the article, but this seems correct.
If you have a perfect model with zero uncertainty, you can solve the entire situation in your head, and then when you actually do it, the result should be the same… or the assumptions were wrong somehow.
Otherwise, I think it makes sense to distinguish two types of situations:
a) During the execution of the plan, something completely unexpected happens. Oops, you have to update, and start thinking again, considering the new information.
b) Your model has some uncertainty, but you know the statistical distributions. For example, with probability 80% the world is in state X, with probability 20% it is in state Y, but you cannot immediately check which option it is. But you can, for example, create a five-step plan based on the (more likely) assumption that it was state X, and if the assumption is wrong, you know it will become visible during step 3, in which case you will switch to alternative steps 4b and 5b. Or if the switch would be too expensive, maybe you could instead add a step zero, some experiment which will figure out whether it is X or Y.
The difference is between “I didn’t expect this—must update and think again” and “I expected this could happen—switching to (already prepared) Plan B”. The former requires iteration, but the latter does not.
An analogy in computer programming would be a) the programmer finding out that the program has a bug and trying to fix it; vs b) the programmer including an “if” statement or exception handling in the program.
In real life the distinction can be less clean. For example, even if you have exact statistical distributions, the resulting number of combinations may be too large to handle computationally, so you might prepare plans for the three (or thirty, if you are a superintelligence) most likely scenarios in advance, and stop and think again if something else happens.
On the other hand, even when unexpected things happen, we often do not immediately freeze and start thinking, but try to stabilize things first. (If I suddenly find during my walk that I am walking in a wrong direction, I will first put both my feet on the ground; and maybe if I am in the middle of a road, I will get to the sidewalk first… and only then I will look at the map and start thinking.) This again assumes that I have correct probability distributions at least about some things (that putting both feet on the ground will increase my stability; that standing on the sidewalk is safer than standing in the middle of a road) even if I got something else wrong (the street I am currently on).
Your model has some uncertainty, but you know the statistical distributions. For example, with probability 80% the world is in state X, with probability 20% it is in state Y.
Nice way of putting it.