An oil refinery is slightly intelligent as an optimization process. Presumably it has sensors that can detect the heaviness of the crude input, its sulfur content, and perhaps other impurities and process it accordingly so that valuable petroleum fractions are maximized. But I don’t know that much about refineries; it may be the case that human operators madly turn valves and pull levers while watching spinning dials to make our gasoline and the refinery is just a big dumb tool.
The question to ask about the refinery as an optimization process is to what degree its initial conditions can be changed and it still produces optimized output. The more powerful it is as an optimizer, the more degrees of freedom on the input it can handle. A perfect refinery could accept waste from a landfill and always produce highly valuable goods.
If you’re asking about future possible states, and literally counting them as Eliezer suggests, even a refinery that has very little input flexibility is a powerful optimizer, provided it puts out well-refined products. It’s a more powerful optimizer than most people, by that metric. How do I know? It has influence over a lot more total mass. The total entropy reduction it performs is vast.
Why does it matter how humans are involved? That’s one of the positive aspects of Eliezer’s definition: it doesn’t care about whether there are humans pulling levers, humans and a lot of automation and sensors, or a fully automated lights-out computerized operation.
The difference in entropy reduction produced doesn’t care about what the refinery could handle, only what it does handle. Switching between different kinds of crude of similar quality won’t change the optimization power rating, even if lots of adjustments get made when that happens. Worse crude, with more weird stuff in it, probably has a marginally higher starting entropy, so using that as feedstock will produce a higher rating. Being able to use it won’t, though.
Using that definition very few things that humans do count as optimization, and virtually nothing our brains do counts. For instance my brain doesn’t till a field and plant seeds and harvest grain; my muscles twitch to make my bones and skin exert forces on a plow and seeds and grains, and the sun and plants do the vast majority of the work. My brain just happens to be the thing that sets the initial process in motion and tweaks it along the way.
Of course maybe since this was just a preliminary article that’s fine, and a study of “intelligence” would focus on the parts of the optimization process that, if removed, would stop it from being an optimization process. I still think, however, that you can’t just take the most favorable starting state and evaluate the results after some period of time, e.g. an unmanned oil refinery with a pipe connected to an infinite reserve of uniform crude oil on one side, a wire with an infinite reserve of electrical potential on another side, and an output connected to an infinitely empty tank for refined petroleum. You have to look at the likely starting states and calculate a probability distribution over the possible final states. In that sense an oil refinery might optimize over a period of a day or so, given current conditions. But current conditions are far from likely; there are trucks (or rail cars, or pipeline contents) on their way to the refinery for tomorrow, and more transport scheduled to empty its product tanks. The refinery, as a single object, is a very limited optimizing process. If placed on any random flat stable area on Earth’s surface it would be nearly useless. Infrastructure matters.
There’s an anthropic or self-referential kind of argument for most things having some optimizing value, however. The likelihood of finding a refinery randomly placed on the Earth or in the Universe is incredibly low, probably lower than the probability of it having zero utility at a random place in the Universe. The more complex an optimizing process is, the more likely it is to be found where it can be most optimal. Something to think about is the conditional probability P(this-process-is-optimizing | this-process-exists). I suspect it’s very close to 1 for very optimal processes. It’s basically the probability of evolution happening, I guess.
Your brain is a causal component of the optimization processes; therefore it seems fair to give it credit. If I take away your plow, it seems reasonable to conclude that your optimization would be less effective, but not ineffective. If I take away your brain, it seems reasonable to conclude that the plow would lose all optimization power. It seems reasonable to conclude from this that your brain has more optimization power, even within that limited context, than the plow does. Sorting out optimization power of your brain vs the plants is difficult for the same reasons that sorting out causality is difficult.
Your point that the definition of “count states” is awkward is exactly the point I’ve been trying to make. Counting states is precisely entropy, which directly implies that refrigerators and oil refineries are powerful optimizers. This conclusion seems problematic, in that it does not align well with all the connotations of “optimization process” that Eliezer is talking about. That’s why I’m saying we need a technical explanation of optimization power, not a loose qualitative explanation.
It seems to me that optimization power should somehow be measured against the complexity of the problem domain. How, I don’t know. I’m just trying to point out that the original post is farther from a complete treatment on the topic than I thought the first time I read it, or than most of the comments seem to give it credit for.
I’m probably better at concrete examples. Consider a list of N comparable items. An optimizing process that orders them from least to greatest (sorts them) preserving relative order has optimization measure 1/N!. A less optimal process by the measure doesn’t maintain relative order and at worst has optimization measure N!/N! (completely reordering a list of N identical items) and at best (N-M)!/N! where M is the number of unique values among the items.
Sorting maintaining order is optimal under the measure. Sorting ignoring order is variable under the measure, depending entirely on the input. I think this where we were getting confused about how to talk about an oil refinery. Clearly an oil refinery is not optimal, so it must be variable. But then what is its measure? Is it the expected value over all possible initial states? Even that may not be unique; consider a list of N items that can take a value from the set {1} versus a list of N items that can take a value from the set {0,1}. Clearly the former is completely unoptimized by the order-ignoring sort, and the latter has expected optimization value (N/2)!/N!. Taking the value of the N items from the natural numbers the process would achieve 1/N! expected optimization. So taking into consideration the complexity of the problem domain won’t help; {1} is simpler than {0,1} is simpler than the natural numbers, but the optimization is inversely related to this progression.
I think this means that optimization is very context dependent. I think it means if I had to send a minimum-length message about the sorting algorithm I would be able to send a shorter message predicting the un-optimal output of sorting a list of N items of value “1” than a message predicting the optimal output of sorting a list of N natural numbers. If I had to gauge the power of the sorting mind I would get different answers depending on the input despite the fact that all sorting algorithms are O(n log n). If I had to gauge the efficiency of sorting any arbitrary output I could clearly say that the order-preserving sort was optimal, but if we simply change the problem domain so that we can no longer recognize relative order of equal values in the list then both algorithms are less optimal since the maximum measure of optimization drops to 1/(N-M)! So it appears that the same physical processes (or algorithms) can be more or less optimal depending on the preferences themselves. This means it may be impossible to infer the actual preferences from the evidence.
An oil refinery is slightly intelligent as an optimization process. Presumably it has sensors that can detect the heaviness of the crude input, its sulfur content, and perhaps other impurities and process it accordingly so that valuable petroleum fractions are maximized. But I don’t know that much about refineries; it may be the case that human operators madly turn valves and pull levers while watching spinning dials to make our gasoline and the refinery is just a big dumb tool.
The question to ask about the refinery as an optimization process is to what degree its initial conditions can be changed and it still produces optimized output. The more powerful it is as an optimizer, the more degrees of freedom on the input it can handle. A perfect refinery could accept waste from a landfill and always produce highly valuable goods.
If you’re asking about future possible states, and literally counting them as Eliezer suggests, even a refinery that has very little input flexibility is a powerful optimizer, provided it puts out well-refined products. It’s a more powerful optimizer than most people, by that metric. How do I know? It has influence over a lot more total mass. The total entropy reduction it performs is vast.
Why does it matter how humans are involved? That’s one of the positive aspects of Eliezer’s definition: it doesn’t care about whether there are humans pulling levers, humans and a lot of automation and sensors, or a fully automated lights-out computerized operation.
The difference in entropy reduction produced doesn’t care about what the refinery could handle, only what it does handle. Switching between different kinds of crude of similar quality won’t change the optimization power rating, even if lots of adjustments get made when that happens. Worse crude, with more weird stuff in it, probably has a marginally higher starting entropy, so using that as feedstock will produce a higher rating. Being able to use it won’t, though.
Using that definition very few things that humans do count as optimization, and virtually nothing our brains do counts. For instance my brain doesn’t till a field and plant seeds and harvest grain; my muscles twitch to make my bones and skin exert forces on a plow and seeds and grains, and the sun and plants do the vast majority of the work. My brain just happens to be the thing that sets the initial process in motion and tweaks it along the way.
Of course maybe since this was just a preliminary article that’s fine, and a study of “intelligence” would focus on the parts of the optimization process that, if removed, would stop it from being an optimization process. I still think, however, that you can’t just take the most favorable starting state and evaluate the results after some period of time, e.g. an unmanned oil refinery with a pipe connected to an infinite reserve of uniform crude oil on one side, a wire with an infinite reserve of electrical potential on another side, and an output connected to an infinitely empty tank for refined petroleum. You have to look at the likely starting states and calculate a probability distribution over the possible final states. In that sense an oil refinery might optimize over a period of a day or so, given current conditions. But current conditions are far from likely; there are trucks (or rail cars, or pipeline contents) on their way to the refinery for tomorrow, and more transport scheduled to empty its product tanks. The refinery, as a single object, is a very limited optimizing process. If placed on any random flat stable area on Earth’s surface it would be nearly useless. Infrastructure matters.
There’s an anthropic or self-referential kind of argument for most things having some optimizing value, however. The likelihood of finding a refinery randomly placed on the Earth or in the Universe is incredibly low, probably lower than the probability of it having zero utility at a random place in the Universe. The more complex an optimizing process is, the more likely it is to be found where it can be most optimal. Something to think about is the conditional probability P(this-process-is-optimizing | this-process-exists). I suspect it’s very close to 1 for very optimal processes. It’s basically the probability of evolution happening, I guess.
Your brain is a causal component of the optimization processes; therefore it seems fair to give it credit. If I take away your plow, it seems reasonable to conclude that your optimization would be less effective, but not ineffective. If I take away your brain, it seems reasonable to conclude that the plow would lose all optimization power. It seems reasonable to conclude from this that your brain has more optimization power, even within that limited context, than the plow does. Sorting out optimization power of your brain vs the plants is difficult for the same reasons that sorting out causality is difficult.
Your point that the definition of “count states” is awkward is exactly the point I’ve been trying to make. Counting states is precisely entropy, which directly implies that refrigerators and oil refineries are powerful optimizers. This conclusion seems problematic, in that it does not align well with all the connotations of “optimization process” that Eliezer is talking about. That’s why I’m saying we need a technical explanation of optimization power, not a loose qualitative explanation.
It seems to me that optimization power should somehow be measured against the complexity of the problem domain. How, I don’t know. I’m just trying to point out that the original post is farther from a complete treatment on the topic than I thought the first time I read it, or than most of the comments seem to give it credit for.
I’m probably better at concrete examples. Consider a list of N comparable items. An optimizing process that orders them from least to greatest (sorts them) preserving relative order has optimization measure 1/N!. A less optimal process by the measure doesn’t maintain relative order and at worst has optimization measure N!/N! (completely reordering a list of N identical items) and at best (N-M)!/N! where M is the number of unique values among the items.
Sorting maintaining order is optimal under the measure. Sorting ignoring order is variable under the measure, depending entirely on the input. I think this where we were getting confused about how to talk about an oil refinery. Clearly an oil refinery is not optimal, so it must be variable. But then what is its measure? Is it the expected value over all possible initial states? Even that may not be unique; consider a list of N items that can take a value from the set {1} versus a list of N items that can take a value from the set {0,1}. Clearly the former is completely unoptimized by the order-ignoring sort, and the latter has expected optimization value (N/2)!/N!. Taking the value of the N items from the natural numbers the process would achieve 1/N! expected optimization. So taking into consideration the complexity of the problem domain won’t help; {1} is simpler than {0,1} is simpler than the natural numbers, but the optimization is inversely related to this progression.
I think this means that optimization is very context dependent. I think it means if I had to send a minimum-length message about the sorting algorithm I would be able to send a shorter message predicting the un-optimal output of sorting a list of N items of value “1” than a message predicting the optimal output of sorting a list of N natural numbers. If I had to gauge the power of the sorting mind I would get different answers depending on the input despite the fact that all sorting algorithms are O(n log n). If I had to gauge the efficiency of sorting any arbitrary output I could clearly say that the order-preserving sort was optimal, but if we simply change the problem domain so that we can no longer recognize relative order of equal values in the list then both algorithms are less optimal since the maximum measure of optimization drops to 1/(N-M)! So it appears that the same physical processes (or algorithms) can be more or less optimal depending on the preferences themselves. This means it may be impossible to infer the actual preferences from the evidence.