Because it serves as a good example, simply put. It gets the idea clear across about what it means, even if there are certainly complexities in comparing evolution to the output of an SGD-trained neural network.
It predicts learning correlates of the reward signal that break apart outside of the typical environment.
When you look at the actual process for how we actually start to like ice-cream—namely, we eat it, and then we get a reward, and that’s why we like it—then the world looks a a lot less hostile, and misalignment a lot less likely.
Yes, that’s why we like it, and that is a way we’re misaligned with evolution (in the ‘do things that end up with vast quantities of our genes everywhere’ sense).
Our taste buds react to it, and they were selected for activating on foods which typically contained useful nutrients, and now they don’t in reality since ice-cream is probably not good for you. I’m not sure what this example is gesturing at? It sounds like a classic issue of having a reward function (‘reproduction’) that ends up with an approximation (‘your tastebuds’) that works pretty well in your ‘training environment’ but diverges in wacky ways outside of that.
I’m inferring by ‘evolution is only selecting hyperparameters’ is that SGD has less layers of indirection between it and the actual operation of the mind compared to evolution (which has to select over the genome which unfolds into the mind). Sure, that gives some reason to believe it will be easier to direct it in some ways—though I think there’s still active room for issues of in-life learning, I don’t really agree with Quintin’s idea that the cultural/knowledge-transfer boom with humans has happened thus AI won’t get anything like it—but even if we have more direct optimization I don’t see that as strongly making misalignment less likely? It does make it somewhat less likely, though it still has many large issues for deciding what reward signals to use.
I still expect correlates of the true objective to be learned, which even in-life training for humans have happen to them through sometimes associating not-related-thing to them getting a good-thing and not just as a matter of false beliefs. Like, as a simple example, learning to appreciate rainy days because you and your family sat around the fire and had fun, such that you later in life prefer rainy days even without any of that.
Evolution doesn’t directly grow minds, but it does directly select for the pieces that grow minds, and has been doing that for quite some time. There’s a reason why it didn’t select for tastebuds that gave a reward signal strictly when some other bacteria in the body reported that they would benefit from it: that’s more complex (to select for), opens more room for ‘bad reporting’, may have problems with shorter gut bacteria lifetimes(?), and a simpler tastebud solution captured most of what it needed!
The way he’s using the example of evolution is captured entirely by that, quite directly, and I don’t find it objectionable.
Because it serves as a good example, simply put. It gets the idea clear across about what it means, even if there are certainly complexities in comparing evolution to the output of an SGD-trained neural network.
It predicts learning correlates of the reward signal that break apart outside of the typical environment.
Yes, that’s why we like it, and that is a way we’re misaligned with evolution (in the ‘do things that end up with vast quantities of our genes everywhere’ sense). Our taste buds react to it, and they were selected for activating on foods which typically contained useful nutrients, and now they don’t in reality since ice-cream is probably not good for you. I’m not sure what this example is gesturing at? It sounds like a classic issue of having a reward function (‘reproduction’) that ends up with an approximation (‘your tastebuds’) that works pretty well in your ‘training environment’ but diverges in wacky ways outside of that.
I’m inferring by ‘evolution is only selecting hyperparameters’ is that SGD has less layers of indirection between it and the actual operation of the mind compared to evolution (which has to select over the genome which unfolds into the mind). Sure, that gives some reason to believe it will be easier to direct it in some ways—though I think there’s still active room for issues of in-life learning, I don’t really agree with Quintin’s idea that the cultural/knowledge-transfer boom with humans has happened thus AI won’t get anything like it—but even if we have more direct optimization I don’t see that as strongly making misalignment less likely? It does make it somewhat less likely, though it still has many large issues for deciding what reward signals to use.
I still expect correlates of the true objective to be learned, which even in-life training for humans have happen to them through sometimes associating not-related-thing to them getting a good-thing and not just as a matter of false beliefs. Like, as a simple example, learning to appreciate rainy days because you and your family sat around the fire and had fun, such that you later in life prefer rainy days even without any of that.
Evolution doesn’t directly grow minds, but it does directly select for the pieces that grow minds, and has been doing that for quite some time. There’s a reason why it didn’t select for tastebuds that gave a reward signal strictly when some other bacteria in the body reported that they would benefit from it: that’s more complex (to select for), opens more room for ‘bad reporting’, may have problems with shorter gut bacteria lifetimes(?), and a simpler tastebud solution captured most of what it needed! The way he’s using the example of evolution is captured entirely by that, quite directly, and I don’t find it objectionable.