It would be interesting if someone discovered something like “junk DNA that just copies itself” within the weights during the backprop+SGD process. Would be some evidence that backprop’s thumb is not so heavy a worm can’t wiggle out. Right now I would bet against that happening within a normal neural net training on a dataset.
Note that RL exists and gives the neural net much more uh “creative room” to uh “decide how to exist”. Because you just have to get enough score over time to survive, but any strategy is accepted. In other words, it is much less convergent.
Also in RL, glitching/hacking of the physics sim / game engine is what you expect to happen! Then you have to patch your sim and retrain.
Also, most of the ML systems we use every day involve multiple neural nets with different goals (eg the image generator and the NSFW detector), so something odd might happen in that interaction.
All this to say: The question “if I train one NN on a fixed dataset with backprop+SGD, could something unexpected pop out?” is quite interesting and still open in my opinion. But even if that always goes exactly as expected, it is certainly clear that RL, active learning, multi-NN ML systems, hyperparameter optimization (which is often an evolutionary algorithm), etc produces weird things with weird goals and strategies very often.
I think debate surrounds the 1-NN-1-dataset question because it is an interesting and natural and important question, the type of question a good scientist would ask. Probably only a small part of the bigger challenge to control the whole trained machine.
It would be interesting if someone discovered something like “junk DNA that just copies itself” within the weights during the backprop+SGD process. Would be some evidence that backprop’s thumb is not so heavy a worm can’t wiggle out. Right now I would bet against that happening within a normal neural net training on a dataset.
Note that RL exists and gives the neural net much more uh “creative room” to uh “decide how to exist”. Because you just have to get enough score over time to survive, but any strategy is accepted. In other words, it is much less convergent.
Also in RL, glitching/hacking of the physics sim / game engine is what you expect to happen! Then you have to patch your sim and retrain.
Also, most of the ML systems we use every day involve multiple neural nets with different goals (eg the image generator and the NSFW detector), so something odd might happen in that interaction.
All this to say: The question “if I train one NN on a fixed dataset with backprop+SGD, could something unexpected pop out?” is quite interesting and still open in my opinion. But even if that always goes exactly as expected, it is certainly clear that RL, active learning, multi-NN ML systems, hyperparameter optimization (which is often an evolutionary algorithm), etc produces weird things with weird goals and strategies very often.
I think debate surrounds the 1-NN-1-dataset question because it is an interesting and natural and important question, the type of question a good scientist would ask. Probably only a small part of the bigger challenge to control the whole trained machine.