It’s really important to use neutral, accurate terminology. AFAICT saying ML “selects for” low loss is unconventional. I think MIRI introduced this terminology. And if you have a bunch of intuitions about evolution being relevant to AI alignment and you want people to believe you on that, you can just use the same words for both optimization processes. Regardless of whether the processes share the relevant causal mechanisms, the two processes both “select for”stuff! They can’t be that different, right?
And now the discussion moves on to debating the differences between two assumed-similar processes—does ML have less of an “information bottleneck”? Does that change the “selection pressures”? Urgh.
I think this terminology sucks and I wish it hadn’t been adopted.
Be careful with words. Words shape how you think and your implicit category boundaries. When thinking privately, I often do better by tossing words to the side and thinking in terms of how each process works, and then considering what I expect to happen as a result of each process.
EDIT: In AGI Ruin: A List of Lethalities, Eliezer says that evolution had a “loss function” when he portrayed ML as similar to evolution:
See, again, the case of human intelligence. We didn’t break alignment with the ‘inclusive reproductive fitness’ outer loss function
And (I don’t immediately know where the comment is), gwern has described evolution as having a “reward function.” This is sloppy shorthand at best, and misleading + distinction-blurring at worst.
It depends on what I’m trying to communicate. For example:
“ML selects for low loss” → “Trained networks tend to have low training loss”
This correctly highlights a meaningful correlation (loss tends to be low for trained networks) and alludes to a relevant mechanism (networks are in fact updated to locally decrease loss on their training distributions). However, it avoids implying that the mechanism of ML is “selection on low loss.”
I think the term is very reasonable and basically accurate, even more so with regard to most RL methods. It’s a good way of describing a training process without implying that the evolving system will head toward optimality deliberately. I don’t know a better way to communicate this succinctly, especially while not being specific about what local search algorithm is being used.
Also, evolutionary algorithms can be used to approximate gradient descent (with noisier gradient estimates), so it’s not unreasonable to use similar language about both.
I’m not a huge fan of the way you imply that it was chosen for rhetorical purposes.
I’m not a huge fan of the way you imply that it was chosen for rhetorical purposes.
To be clear, I’m not alleging mal-intent or anything. I’m more pointing out memetic dynamics. The situation can look as innocent as “You genuinely believe X, and think it’s important for people to get X, and so you iterate over explanations until you find an effective one.” And maybe that explanation just happens to involve analogizing that ML “selects for low loss.”
(I don’t mean to dogpile) I think that selection is the correct word, and that it doesn’t really seem to be smuggling in incorrect connections to evolution.
We could imagine finding a NN that does well according to a loss function by simply randomly initializing many many NNs, and then keeping the one that does best according to the loss function. I think this process would accurately be described as selection; we are literally selecting the model which does best.
I’m not claiming that SGD does this[1], just giving an example of a method to find a low-loss parameter configuration which isn’t related to evolution, and is (in my opinion) best described as “selection”.
Sure. But I think that’s best described as “best-of-k sampling”, which is still better because it avoids implicitly comparing selection-over-learning-setups (i.e. genotypes) with selection-over-parameterizations.
But let’s just say I concede this particular method can be non-crazily called “selection.” AFAICT I think you’re arguing: “There exist ML variants which can be described as ‘selection’.” But speculation about “selecting for low loss” is not confined to those variants, usually people just lump everything in as that. And I doubt that most folks are on the edge of their seats, ready to revoke the analogy if some paper comes out that convincingly shows that ML diverges from “selecting for low loss”...[1]
Actually, I agreed too quickly. Words are not used in a vacuum. Even though this method isn’t related to evolution, and even though a naive person might call it “selection” (and have that be descriptively reasonable), that doesn’t mean it’s best described as “selection.” The reason is that the “s-word” has lots of existing evolutionary connotations. And on my understanding, that’s the main reason you want to call it “selection” to begin with—in order to make analogical claims about the results of this process compared to the results of evolution.
But my whole point is that the analogy is only valid if the two optimization processes (evolution and best-of-k sampling) share the relevant causal mechanisms. So before you start using the s-word and especially before you start using its status as “selection” to support analogies, I want to see that argument first. Else, it should be called something more neutral.
I suggest interpreting phenomenon as multi-level nested optimization paradigm: many systems can be usefully described as having two (or more) levels where a slow sample-inefficient but ground-truth ‘outer’ loss such as death, bankruptcy, or reproductive fitness, trains & constrains a fast sample-efficient but possibly misguided ‘inner’ loss which is used by learned mechanisms such as neural networks or linear programming group selection perspective.
There is more to the post though! I recommend reading it, especially if you’re confused what this could possibly concretely mean when all natural selection is is an update process, and no real outer loss is defined. Especially the section Two-Level Meta Learning. I do not think he makes this mistake out of naivete.
It’s really important to use neutral, accurate terminology. AFAICT saying ML “selects for” low loss is unconventional. I think MIRI introduced this terminology. And if you have a bunch of intuitions about evolution being relevant to AI alignment and you want people to believe you on that, you can just use the same words for both optimization processes. Regardless of whether the processes share the relevant causal mechanisms, the two processes both “select for” stuff! They can’t be that different, right?
And now the discussion moves on to debating the differences between two assumed-similar processes—does ML have less of an “information bottleneck”? Does that change the “selection pressures”? Urgh.
I think this terminology sucks and I wish it hadn’t been adopted.
Be careful with words. Words shape how you think and your implicit category boundaries. When thinking privately, I often do better by tossing words to the side and thinking in terms of how each process works, and then considering what I expect to happen as a result of each process.
See also: Think carefully before calling RL policies “agents”.
EDIT: In AGI Ruin: A List of Lethalities, Eliezer says that evolution had a “loss function” when he portrayed ML as similar to evolution:
And (I don’t immediately know where the comment is), gwern has described evolution as having a “reward function.” This is sloppy shorthand at best, and misleading + distinction-blurring at worst.
What’s your preferred terminology?
It depends on what I’m trying to communicate. For example:
“ML selects for low loss” → “Trained networks tend to have low training loss”
This correctly highlights a meaningful correlation (loss tends to be low for trained networks) and alludes to a relevant mechanism (networks are in fact updated to locally decrease loss on their training distributions). However, it avoids implying that the mechanism of ML is “selection on low loss.”
I think the term is very reasonable and basically accurate, even more so with regard to most RL methods. It’s a good way of describing a training process without implying that the evolving system will head toward optimality deliberately. I don’t know a better way to communicate this succinctly, especially while not being specific about what local search algorithm is being used.
Also, evolutionary algorithms can be used to approximate gradient descent (with noisier gradient estimates), so it’s not unreasonable to use similar language about both.
I’m not a huge fan of the way you imply that it was chosen for rhetorical purposes.
Without commenting on the rest for now—
To be clear, I’m not alleging mal-intent or anything. I’m more pointing out memetic dynamics. The situation can look as innocent as “You genuinely believe X, and think it’s important for people to get X, and so you iterate over explanations until you find an effective one.” And maybe that explanation just happens to involve analogizing that ML “selects for low loss.”
(I don’t mean to dogpile)
I think that selection is the correct word, and that it doesn’t really seem to be smuggling in incorrect connections to evolution.
We could imagine finding a NN that does well according to a loss function by simply randomly initializing many many NNs, and then keeping the one that does best according to the loss function. I think this process would accurately be described as selection; we are literally selecting the model which does best.
I’m not claiming that SGD does this[1], just giving an example of a method to find a low-loss parameter configuration which isn’t related to evolution, and is (in my opinion) best described as “selection”.
Although “Is SGD a Bayesian sampler? Well, almost” does make a related claim.
Sure. But I think that’s best described as “best-of-k sampling”, which is still better because it avoids implicitly comparing selection-over-learning-setups (i.e. genotypes) with selection-over-parameterizations.
But let’s just say I concede this particular method can be non-crazily called “selection.” AFAICT I think you’re arguing: “There exist ML variants which can be described as ‘selection’.” But speculation about “selecting for low loss” is not confined to those variants, usually people just lump everything in as that. And I doubt that most folks are on the edge of their seats, ready to revoke the analogy if some paper comes out that convincingly shows that ML diverges from “selecting for low loss”...[1]
To be clear, that evidence already exists.
Hi, do you have a links to the papers/evidence?
Actually, I agreed too quickly. Words are not used in a vacuum. Even though this method isn’t related to evolution, and even though a naive person might call it “selection” (and have that be descriptively reasonable), that doesn’t mean it’s best described as “selection.” The reason is that the “s-word” has lots of existing evolutionary connotations. And on my understanding, that’s the main reason you want to call it “selection” to begin with—in order to make analogical claims about the results of this process compared to the results of evolution.
But my whole point is that the analogy is only valid if the two optimization processes (evolution and best-of-k sampling) share the relevant causal mechanisms. So before you start using the s-word and especially before you start using its status as “selection” to support analogies, I want to see that argument first. Else, it should be called something more neutral.
Gwern talks about natural selection like it has a loss function in Evolution as Backstop For Reinforcement Learning:
There is more to the post though! I recommend reading it, especially if you’re confused what this could possibly concretely mean when all natural selection is is an update process, and no real outer loss is defined. Especially the section Two-Level Meta Learning. I do not think he makes this mistake out of naivete.
perhaps ml doesn’t “select for” a loss function, it “pulls towards” a loss function.