Because the comment assumes that all these brilliant people on the new team would interpret “novelty-seeking” in a very straightforward (and, actually, quite boring) way: “keep rearranging the universe into configurations it hasn’t seen before, as fast as it possibly can”.
If any of us can rearrange things as fast as one possible can, that person would get bored within hours (if not minutes).
The people doing that project will ponder what makes life interesting and will try to formalize that… This is a very strong team (judging from the listing of names in the post), they will figure out something creative.
That being said, safety challenges in that approach are formidable. The most curious thing one can do is probably to self-modify in various interesting ways and see how it feels (not as fast as possible, and not quite in arbitrary ways, but still to explore plenty of variety). So one would need to explicitly address all safety issues associated with open-ended recursive self-modification. It’s not easy at all...
The word “curiosity” has a fairly well-defined meaning in the Reinforcement Learning literature (see for instance this paper). There are vast numbers of papers that try to come up with ways to give an agent intrinsic rewards that map onto the human understanding of “curiosity”, and almost all of them are some form of “go towards states you haven’t seen before”. The predictable consequence of prioritising states you haven’t seen before is that you will want to change the state of the universe very very quickly.
Novelty is important. Going towards states you have not seen before is important. This will be a part of the new system, that’s for sure.
But this team is under no obligation to follow whatever current consensus might be (if there is a consensus). Whatever is the state of the field, it can’t claim a monopoly on how words “curiosity” or “novelty” are interpreted, what are the good ways to maximize them… How one constrains going through a subset of all those novel states by aesthetics, by the need to take time and enjoy (“exploit”) those new states, and by safety considerations (so, by predicting whether the novel state will be useful and not detrimental)… All this will be on the table...
Some of the people on this team are known for making radical breakthroughs in machine learning and for founding new subfields in machine learning. They are not going to blindly copy the approaches from the existing literature (although they will take existing literature into account).
Because the comment assumes that all these brilliant people on the new team would interpret “novelty-seeking” in a very straightforward (and, actually, quite boring) way: “keep rearranging the universe into configurations it hasn’t seen before, as fast as it possibly can”.
If any of us can rearrange things as fast as one possible can, that person would get bored within hours (if not minutes).
The people doing that project will ponder what makes life interesting and will try to formalize that… This is a very strong team (judging from the listing of names in the post), they will figure out something creative.
That being said, safety challenges in that approach are formidable. The most curious thing one can do is probably to self-modify in various interesting ways and see how it feels (not as fast as possible, and not quite in arbitrary ways, but still to explore plenty of variety). So one would need to explicitly address all safety issues associated with open-ended recursive self-modification. It’s not easy at all...
The word “curiosity” has a fairly well-defined meaning in the Reinforcement Learning literature (see for instance this paper). There are vast numbers of papers that try to come up with ways to give an agent intrinsic rewards that map onto the human understanding of “curiosity”, and almost all of them are some form of “go towards states you haven’t seen before”. The predictable consequence of prioritising states you haven’t seen before is that you will want to change the state of the universe very very quickly.
Novelty is important. Going towards states you have not seen before is important. This will be a part of the new system, that’s for sure.
But this team is under no obligation to follow whatever current consensus might be (if there is a consensus). Whatever is the state of the field, it can’t claim a monopoly on how words “curiosity” or “novelty” are interpreted, what are the good ways to maximize them… How one constrains going through a subset of all those novel states by aesthetics, by the need to take time and enjoy (“exploit”) those new states, and by safety considerations (so, by predicting whether the novel state will be useful and not detrimental)… All this will be on the table...
Some of the people on this team are known for making radical breakthroughs in machine learning and for founding new subfields in machine learning. They are not going to blindly copy the approaches from the existing literature (although they will take existing literature into account).