I love the example, I’d never heard of that project before.
I’m agnostic on demonic intelligence. I think the key point is not the demons themselves but the process which produces them. Somehow, an imperfect optimizing search process induces a secondary optimizer, and it’s that secondary optimizer which produces the demons. For instance, in the metabolism example, evolution is the secondary optimizer, and its goals are (often) directly opposed to the original optimizer—it wants to conserve free energy, in order to “trade” with the free energy optimizer later. The demons themselves (i.e. cells/enzymes in the metabolism example) are inner optimizers of the secondary optimizer; I expect that Risks From Learned Optimization already describes the secondary optimizer <-> demon relationship fairly well, including when the demons will be more/less intelligent.
The interesting/scary point is that the secondary optimizer is consistently opposed to the original optimizer; the two are basically playing a game where the secondary tries to hide information from the original.
Hmmm, this doesn’t work to distinguish the two for me. Couldn’t you say a local minima involves a secondary optimizing search process that has that minima as its objective? To use your ball analogy, what exactly is the difference between these twisty demon hills and a simple crater-shaped pit? (Or, what is the difference between a search process that is vulnerable to twisty demon hills and one which is vulnerable to pits?)
In the ball example, it’s the selection process that’s interesting—the ball ending up rolling alongside one bump or another, and bumps “competing” in the sense that the ball will eventually end up rolling along at most one of them (assuming they run in different directions).
Couldn’t you say a local minima involves a secondary optimizing search process that has that minima as its objective?
Only if such a search process is actually taking place. That’s why it’s key to look at the process, rather than the bumps and valleys themselves.
To use your ball analogy, what exactly is the difference between these twisty demon hills and a simple crater-shaped pit?
There isn’t inherently any important difference between those two. That said, there are some environments in which “bumps” which effectively steer a ball will tend to continue to do so in the future, and other environments in which the whole surface is just noise with low spatial correlation. The latter would not give rise to demons (I think), while the former would. This is part of what I’m still confused about—what, quantitatively, are the properties of the environment necessary for demons to show up?
Does that help clarify, or should I take another stab at it?
Ah, that does help, thanks. In my words: A search process that is vulnerable to local minima doesn’t necessarily contain a secondary search process, because it might not be systematically comparing local minima and choosing between them according to some criteria. It just goes for the first one it falls for, or maybe slightly more nuanced, the first sufficiently big one it falls for.
By contrast, in the ball rolling example you gave, the walls/ridges were competing with each other, such that the “best” one (or something like that) would be systematically selected by the ball, rather than just the first one or the first-sufficiently-big one.
So in that case, looking over your list again...
OK, I think I see how organic life arising from chemistry is an example of a secondary search process. It’s not just a local minima that chemistry found itself in, it’s a big competition between different kinds of local minima. And now I think I see how this would go in the other examples too. As I originally said in my top-level comment, I’m not sure this applies to the example I brought up, actually. Would the “Insert my name as the author of all useful heuristics” heuristic be outcompeted by something else eventually, or not? I bet not, which indicates that it’s a “mere” local minima and not one that is part of a broader secondary search process.
+1, creating a self-reinforcing feedback loop =/= being an optimiser, and so I think any explanation of demons needs to focus on them making deliberate choices to reinforce themselves.
I love the example, I’d never heard of that project before.
I’m agnostic on demonic intelligence. I think the key point is not the demons themselves but the process which produces them. Somehow, an imperfect optimizing search process induces a secondary optimizer, and it’s that secondary optimizer which produces the demons. For instance, in the metabolism example, evolution is the secondary optimizer, and its goals are (often) directly opposed to the original optimizer—it wants to conserve free energy, in order to “trade” with the free energy optimizer later. The demons themselves (i.e. cells/enzymes in the metabolism example) are inner optimizers of the secondary optimizer; I expect that Risks From Learned Optimization already describes the secondary optimizer <-> demon relationship fairly well, including when the demons will be more/less intelligent.
The interesting/scary point is that the secondary optimizer is consistently opposed to the original optimizer; the two are basically playing a game where the secondary tries to hide information from the original.
Hmmm, this doesn’t work to distinguish the two for me. Couldn’t you say a local minima involves a secondary optimizing search process that has that minima as its objective? To use your ball analogy, what exactly is the difference between these twisty demon hills and a simple crater-shaped pit? (Or, what is the difference between a search process that is vulnerable to twisty demon hills and one which is vulnerable to pits?)
In the ball example, it’s the selection process that’s interesting—the ball ending up rolling alongside one bump or another, and bumps “competing” in the sense that the ball will eventually end up rolling along at most one of them (assuming they run in different directions).
Only if such a search process is actually taking place. That’s why it’s key to look at the process, rather than the bumps and valleys themselves.
There isn’t inherently any important difference between those two. That said, there are some environments in which “bumps” which effectively steer a ball will tend to continue to do so in the future, and other environments in which the whole surface is just noise with low spatial correlation. The latter would not give rise to demons (I think), while the former would. This is part of what I’m still confused about—what, quantitatively, are the properties of the environment necessary for demons to show up?
Does that help clarify, or should I take another stab at it?
Ah, that does help, thanks. In my words: A search process that is vulnerable to local minima doesn’t necessarily contain a secondary search process, because it might not be systematically comparing local minima and choosing between them according to some criteria. It just goes for the first one it falls for, or maybe slightly more nuanced, the first sufficiently big one it falls for.
By contrast, in the ball rolling example you gave, the walls/ridges were competing with each other, such that the “best” one (or something like that) would be systematically selected by the ball, rather than just the first one or the first-sufficiently-big one.
So in that case, looking over your list again...
OK, I think I see how organic life arising from chemistry is an example of a secondary search process. It’s not just a local minima that chemistry found itself in, it’s a big competition between different kinds of local minima. And now I think I see how this would go in the other examples too. As I originally said in my top-level comment, I’m not sure this applies to the example I brought up, actually. Would the “Insert my name as the author of all useful heuristics” heuristic be outcompeted by something else eventually, or not? I bet not, which indicates that it’s a “mere” local minima and not one that is part of a broader secondary search process.
+1, creating a self-reinforcing feedback loop =/= being an optimiser, and so I think any explanation of demons needs to focus on them making deliberate choices to reinforce themselves.