Model-based RL algorithms tend to create agents with a big mess of desires and aversions that are not necessarily self-consistent. (But see caveat about self-modification below.)
When an agent is in an environment wherein its desires can change (e.g. an RL agent being updated online by TD learning), it will tend to take foresighted actions to preserve its current desires (cf. instrumental convergence), assuming the agent is sufficiently self-aware and foresighted.
I disagree with how this post uses the word “values” throughout, rather than “desires” (or “preferences”) which (AFAICT) would be a better match to how the term is being used here.
I disagree with how this post seems to optimistically ignore the possibility that the AGI might self-modify to be more coherent in a way that involves crushing / erasing a subset of its desires, and this subset might include the desires related to human flourishing. This is analogous to how you or I might try to self-modify to erase some of our (unendorsed) desires (to eat junk food, be selfish, be cruel, etc.), if we could. (This doesn’t even require exotic things like access to source code / brain tissue; there are lots of mundane tricks to kick bad habits etc.) (Cf. also how people will “bite the bullet” in thought experiments when doing moral reasoning.)
Counterpoint: What if we try to ensure that the AGI has a strong (meta-)preference not to do that?
Response to counterpoint: Sure! That seems like a promising thing to look into. But until we have a plan to ensure that that actually happens, I’ll keep feeling like this is an “area that warrants further research” not “cause for general optimism”.
Relatedly and more broadly, my attitude to the technical alignment problem is more like “It is as yet unclear whether or not we face certain doom if we train a model-based RL agent to AGI using the best practices that AI alignment researchers currently know about” (see §14.6 here), not “We have strong reason to believe that the problem is solvable this way with no big new alignment ideas”. This makes me “optimistic” about solvability of technical alignment by the standards of, say, Eliezer, but not “optimistic” as the term is normally used. More like “uncertain”.
I guess I’m not sure if this is really a disagreement with the post. The post seems to have optimistic vibes in certain places (“if shard theory is true, meaningful partial alignment successes are possible”) but the conclusion is more cautious. ¯\_(ツ)_/¯
I think some of the detailed descriptions of (anthropomorphized) shards are misleading, but I’m not sure that really matters for anything.
Thanks as always for your consistently thoughtful comments:)
I disagree with how this post seems to optimistically ignore the possibility that the AGI might self-modify to be more coherent in a way that involves crushing / erasing a subset of its desires, and this subset might include the desires related to human flourishing.
I also feel this is an “area that warrants further research”, though I don’t view shard-coordination as being different than shard formation. If you understand how inner-values form from outer reward schedules, then how inner-values interact is also a steerable reinforcement. Though this may be exactly what you meant by “try to ensure that the AGI has a strong (meta-)preference not to do that”, so the only disagreement is on the optimism vibe?
I don’t view shard-coordination as being different than shard formation
Yeah I expect that the same learning algorithm source code would give rise to both preferences and meta-preferences. (I think that’s what you’re saying there right?)
From the perspective of sculpting AGI motivations, I think it might be trickier to directly intervene on meta-preferences than to directly intervene on (object-level) preferences, because if the AGI is attending to something related to sensory input, you can kinda guess what it’s probably thinking about and you at least have a chance of issuing appropriate rewards by doing obvious straightforward things, whereas if the AGI is introspecting on its own current preferences, you need powerful interpretability techniques to even have a chance to issue appropriate rewards, I suspect. That’s not to say it’s impossible! We should keep thinking about it. It’s very much on my own mind, see e.g. my silly tweets from just last night.
the AGI might self-modify to be more coherent in a way that involves crushing / erasing a subset of its desires, and this subset might include the desires related to human flourishing. This is analogous to how you or I might try to self-modify to erase some of our (unendorsed) desires (to eat junk food, be selfish, be cruel, etc.), if we could
This seems like a big deal to me, because it feels like if I could modify myself, I would probably do so to make myself better at achieving a handful of goals like {having a positive impact, (maybe) obtaining a large amount of power/money, getting really really good at a particular skill} and everything else like {desire to eat good food, be vengeful, etc} would be thrown out. The first set feels different from the second because those desires feel more like maximising something, which is worrying.
I disagree with how this post uses the word “values” throughout, rather than “desires” (or “preferences”) which (AFAICT) would be a better match to how the term is being used here.
This has definitely been a point of confusion. There are a couple of ways one might reasonably interpret the phrase “human values”:
the common denominator between all humans ever about what they care about
the ethical consensus of (some subset of) humanity
the injunctions a particular human would verbally endorse
the cognitive artifacts inside each particular human that implement that human’s valuing-of-X, including the cases where they verbally endorse that valuing (along with a bunch of other kinds of preferences, both wanted and unwanted)
I think the shard theory workstream generally uses “human values” in the last sense.
I view values as action-determinants, the moment-to-moment internal tugs which steer my thinking and action. I cash out “values”/”subshards” as “contextually activated computations which are shaped into existence by past reinforcement, and which often steer towards their historical reinforcers.”
This is a considerably wider definition of “human values” than usually considered. For example, I might have a narrowly activated value against taking COVID tests, because past reinforcement slapped down my decision to do so after I tested positive and realized I’d be isolating all alone. (The testing was part of why I had to isolate, after all, so I infer that my credit assignment tagged that decision for down-weighting)
This unusual definitional broadness is a real cost, which is why I like talking about an anti-COVID-test subshard, instead of an anti-test “value.”
Things I agree with:
Model-based RL algorithms tend to create agents with a big mess of desires and aversions that are not necessarily self-consistent. (But see caveat about self-modification below.)
When an agent is in an environment wherein its desires can change (e.g. an RL agent being updated online by TD learning), it will tend to take foresighted actions to preserve its current desires (cf. instrumental convergence), assuming the agent is sufficiently self-aware and foresighted.
Human within-lifetime learning is an example of model-based RL, and well worth looking into in the course of trying to understand more generally how powerful model-based RL agents might behave.
Things I disagree with:
I disagree with how this post uses the word “values” throughout, rather than “desires” (or “preferences”) which (AFAICT) would be a better match to how the term is being used here.
I disagree with how this post seems to optimistically ignore the possibility that the AGI might self-modify to be more coherent in a way that involves crushing / erasing a subset of its desires, and this subset might include the desires related to human flourishing. This is analogous to how you or I might try to self-modify to erase some of our (unendorsed) desires (to eat junk food, be selfish, be cruel, etc.), if we could. (This doesn’t even require exotic things like access to source code / brain tissue; there are lots of mundane tricks to kick bad habits etc.) (Cf. also how people will “bite the bullet” in thought experiments when doing moral reasoning.)
Counterpoint: What if we try to ensure that the AGI has a strong (meta-)preference not to do that?
Response to counterpoint: Sure! That seems like a promising thing to look into. But until we have a plan to ensure that that actually happens, I’ll keep feeling like this is an “area that warrants further research” not “cause for general optimism”.
Relatedly and more broadly, my attitude to the technical alignment problem is more like “It is as yet unclear whether or not we face certain doom if we train a model-based RL agent to AGI using the best practices that AI alignment researchers currently know about” (see §14.6 here), not “We have strong reason to believe that the problem is solvable this way with no big new alignment ideas”. This makes me “optimistic” about solvability of technical alignment by the standards of, say, Eliezer, but not “optimistic” as the term is normally used. More like “uncertain”.
I guess I’m not sure if this is really a disagreement with the post. The post seems to have optimistic vibes in certain places (“if shard theory is true, meaningful partial alignment successes are possible”) but the conclusion is more cautious. ¯\_(ツ)_/¯
I think some of the detailed descriptions of (anthropomorphized) shards are misleading, but I’m not sure that really matters for anything.
Thanks as always for your consistently thoughtful comments:)
I also feel this is an “area that warrants further research”, though I don’t view shard-coordination as being different than shard formation. If you understand how inner-values form from outer reward schedules, then how inner-values interact is also a steerable reinforcement. Though this may be exactly what you meant by “try to ensure that the AGI has a strong (meta-)preference not to do that”, so the only disagreement is on the optimism vibe?
Yeah I expect that the same learning algorithm source code would give rise to both preferences and meta-preferences. (I think that’s what you’re saying there right?)
From the perspective of sculpting AGI motivations, I think it might be trickier to directly intervene on meta-preferences than to directly intervene on (object-level) preferences, because if the AGI is attending to something related to sensory input, you can kinda guess what it’s probably thinking about and you at least have a chance of issuing appropriate rewards by doing obvious straightforward things, whereas if the AGI is introspecting on its own current preferences, you need powerful interpretability techniques to even have a chance to issue appropriate rewards, I suspect. That’s not to say it’s impossible! We should keep thinking about it. It’s very much on my own mind, see e.g. my silly tweets from just last night.
This seems like a big deal to me, because it feels like if I could modify myself, I would probably do so to make myself better at achieving a handful of goals like {having a positive impact, (maybe) obtaining a large amount of power/money, getting really really good at a particular skill} and everything else like {desire to eat good food, be vengeful, etc} would be thrown out. The first set feels different from the second because those desires feel more like maximising something, which is worrying.
This has definitely been a point of confusion. There are a couple of ways one might reasonably interpret the phrase “human values”:
the common denominator between all humans ever about what they care about
the ethical consensus of (some subset of) humanity
the injunctions a particular human would verbally endorse
the cognitive artifacts inside each particular human that implement that human’s valuing-of-X, including the cases where they verbally endorse that valuing (along with a bunch of other kinds of preferences, both wanted and unwanted)
I think the shard theory workstream generally uses “human values” in the last sense.
I view values as action-determinants, the moment-to-moment internal tugs which steer my thinking and action. I cash out “values”/”subshards” as “contextually activated computations which are shaped into existence by past reinforcement, and which often steer towards their historical reinforcers.”
This is a considerably wider definition of “human values” than usually considered. For example, I might have a narrowly activated value against taking COVID tests, because past reinforcement slapped down my decision to do so after I tested positive and realized I’d be isolating all alone. (The testing was part of why I had to isolate, after all, so I infer that my credit assignment tagged that decision for down-weighting)
This unusual definitional broadness is a real cost, which is why I like talking about an anti-COVID-test subshard, instead of an anti-test “value.”
Hmm, I think I’d say “I might feel an aversion…” there.
“Desires & aversions” would work in a context where the sign was ambiguous. So would “preferences”.