The reason I often bring up human evolution is because that’s our only example of an outer optimization loop producing an inner general intelligence
There’s also human baby brains training minds from something close to random initialisation at birth into a general intelligence. That example is plausibly a lot closer to how we might expect AGI training to go, because human brains are neural nets too and presumably have strictly-singular flavoured learning dynamics just like our artificial neural networks do. Whereas evolution acts on genes, which to my knowledge don’t have neat NN-style loss landscapes heavily biased towards simplicity.
Evolution is more like if people used classic genetic optimisation to blindly find neural network architectures, optimisers, training losses, and initialisation schemes, that are in turn evaluated by actually training the networks.
Not that I think this ultimately ends up weakening Doomimir’s point all that much. Humans don’t seem to end up with terminal goals that are straightforward copies of the reward circuits pre-wired into our brains either. I sure don’t care much about predicting sensory inputs super accurately, which was probably a very big part of the training signal that build my mind.
you totally care about predicting sensory inputs accurately! maybe mostly instrumentally, but you definitely do? like, what, would it just not bother you at all if you started hallucinating all the time?
Huh. I… think I kind of do care terminally? Or maybe I’m just having a really hard time imagining what it would be like to be terrible at predicting sensory input without this having a bunch of negative consequences.
There’s also human baby brains training minds from something close to random initialisation at birth into a general intelligence. That example is plausibly a lot closer to how we might expect AGI training to go, because human brains are neural nets too and presumably have strictly-singular flavoured learning dynamics just like our artificial neural networks do. Whereas evolution acts on genes, which to my knowledge don’t have neat NN-style loss landscapes heavily biased towards simplicity.
Evolution is more like if people used classic genetic optimisation to blindly find neural network architectures, optimisers, training losses, and initialisation schemes, that are in turn evaluated by actually training the networks.
Not that I think this ultimately ends up weakening Doomimir’s point all that much. Humans don’t seem to end up with terminal goals that are straightforward copies of the reward circuits pre-wired into our brains either. I sure don’t care much about predicting sensory inputs super accurately, which was probably a very big part of the training signal that build my mind.
you totally care about predicting sensory inputs accurately! maybe mostly instrumentally, but you definitely do? like, what, would it just not bother you at all if you started hallucinating all the time?
Instrumentally, yes. The point is that I don’t really care terminally.
Huh. I… think I kind of do care terminally? Or maybe I’m just having a really hard time imagining what it would be like to be terrible at predicting sensory input without this having a bunch of negative consequences.