zhukeepa(Alex Zhu)

Me­taphilo­soph­i­cal com­pe­tence can’t be dis­en­tan­gled from alignment

Refram­ing mis­al­igned AGI’s: well-in­ten­tioned non-neu­rotyp­i­cal assistants

• zhukeepa 1 Apr 2018 1:22 UTC

  4 points

  in reply to: Ben Pace’s comment on: Refram­ing mis­al­igned AGI’s: well-in­ten­tioned non-neu­rotyp­i­cal assistants

  How about now? :P

Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah

My take on agent foun­da­tions: for­mal­iz­ing metaphilo­soph­i­cal competence

• zhukeepa 1 Apr 2018 20:23 UTC

  7 points

  in reply to: quanticle’s comment on: Me­taphilo­soph­i­cal com­pe­tence can’t be dis­en­tan­gled from alignment

  Well, wouldn’t it be great if we had sound metaphilosophical principles that help us distinguish epistemic pits from correct conclusions! :P

  I actually think humanity is in a bunch of epistemic pits that we mostly aren’t even aware of. For example, if you share my view that Buddhist enlightenment carries significant (albeit hard-to-articulate) epistemic content, then basically all of humanity over basically all of time has been in the epistemic pit of non-enlightenment.

  If we figure out the metaphilosophy of how to robustly avoid epistemic pits, and build that into an aligned AGI, then in some sense none of our current epistemic pits are that bad, since that AGI would help us climb out in relatively short order. But if we don’t figure it out, we’ll plausibly stay in our epistemic pits for unacceptably long periods of time.

• zhukeepa 1 Apr 2018 20:47 UTC

  9 points

  on: Me­taphilo­soph­i­cal com­pe­tence can’t be dis­en­tan­gled from alignment

  To test how much my proposed crux is in fact a crux, I’d like for folks to share their intuitions about how many people are metaphilosophically competent enough to safely 1,000,000,000,000,000x, along with their intuitions about the difficulty of AI alignment.

  My current intuition is that there are under 100 people whom, if 1,000,000,000,000,000x’d, would end up avoiding irreversible catastrophes with > 50% probability. (I haven’t thought too much about this question, and wouldn’t be surprised if I update to thinking there are fewer than 10 such people, or even 0 such people.) I also think AI alignment is pretty hard, and necessitates solving difficult metaphilosophical problems.

  Once humanity makes enough metaphilosophical progress (which might require first solving agent foundations), I might feel comfortable 1,000,000,000,000,000x’ing the most metaphilosophically competent person alive, though it’s possible I’ll decide I wouldn’t want to 1,000,000,000,000,000x anyone running on current biological hardware. I’d also feel good 1,000,000,000,000,000x’ing someone if we’re in the endgame and the default outcome is clearly self-annihilation.

  All of these intuitions are weakly held.

• zhukeepa 2 Apr 2018 8:08 UTC

  5 points

  in reply to: Rohin Shah’s comment on: Me­taphilo­soph­i­cal com­pe­tence can’t be dis­en­tan­gled from alignment

  I think most humans achieving what they currently consider their goals would end up being catastrophic for humanity, even if they succeed. (For example I think an eternal authoritarian regime is pretty catastrophic.)

• zhukeepa 2 Apr 2018 8:15 UTC

  2 points

  in reply to: steven0461’s comment on: Me­taphilo­soph­i­cal com­pe­tence can’t be dis­en­tan­gled from alignment

  My model is that giving them the five-minute lecture on the dangers of value lock-in won’t help much. (We’ve tried giving five-minute lectures on the dangers of building superintelligences...) And I think most people executing “find out what would I think if I reflected more, and what the actual causes are of everyone else’s opinions” would get stuck in an epistemic pit and not realize it.

• zhukeepa 2 Apr 2018 19:19 UTC

  5 points

  in reply to: RyanCarey’s comment on: Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah

  Overall, my impression is that you thought I was saying, “A corrigible AI might turn against its operators and kill us all, and the only way to prevent that is by ensuring the AI is metaphilosophically competent.” I was really trying to say “A corrigible AI might not turn against its operators and might not kill us all, and the outcome can still be catastrophic. To prevent this, we’d definitely want our operators to be metaphilosophically competent, and we’d definitely want our AI to not corrupt them. The latter may be simple to ensure if the AI isn’t broadly superhumanly powerful, but may be difficult to ensure if the AI is broadly superhumanly powerful and we don’t have formal guarantees”. My sense is that we actually agree on the latter and agree that the former is wrong. Does this sound right? (I do think it’s concerning that my original post led you to reasonably interpret me as saying the former. Hopefully my edits make this clearer. I suspect part of what happened is that I think humans surviving but producing astronomical moral waste is about as bad as human extinction, so I didn’t bother delineating them, even though this is probably an unusual position.)

  See below for individual responses.

  You omitted a key component of the quote that almost entirely reversis its meaning. The correct quote would read [emphasis added]: “[The AI should] help me figure out whether I built the right AI and correct any mistakes I made, remain informed about the AI’s behavior and avoid unpleasant surprises, make better decisions and clarify my preferences, acquire resources and remain in effective control of them, ensure that my AI systems continue to do all of these nice things...”. i.e. the AI should help with ensuring that the control continues to reside in the human, rather than in itself.

  I edited my post accordingly. This doesn’t change my perspective at all.

  To my understanding, the point of corrigibility is that a corrigible system is supposed to benefit its human operators even if its intentions are somewhat wrong, so it is rather a non sequitur to say that an agent is corrigible because it has the best of intentions in its heart of hearts. If it truly fully understood human intentions and values, corrigibility may even be unnecessary.

  I gave that description to illustrate one way it is like a corrigible agent, which does have the best of intentions (to help its operators), not to imply that a well-intentioned agent is corrigible. I edited it to “In his heart of hearts, the messiah would be trying to help them, and everyone would know that.” Does that make it clearer?

  Clearly you’re right that corrigibility is not sufficient for safety. A corrigible agent can still be instructed by its human operators to make a decision that is irreversibly bad. But it seems to help, and to help a lot. The point of a corrigible AI si that once it takes a few murderous actions, you can switch it off, or tell it to pursue a different objective. So for the messiah example, a corrigible messiah might poison a few followers and then when it is discovered, respond to an instruction to desist. An incorrigible messiah might be asked to stop murdering followers, but continue to do so anyway. So many of the more mundane existential risks would be mitigated by corrigibility.

  I agree that corrigibility helps a bunch with mundane existential risks, and think that e.g. a corrigible misaligned superintelligence is unlikely to lead to self-annihilation, but pretty likely to lead to astronomical moral waste. I edited from “Surely, we wouldn’t build a superintelligence that would guide us down such an insidious path?” to “I don’t think a corrigible superintelligence would guide us down such an insidious path. I even think it would substantially improve the human condition, and would manage to avoid killing us all. But I think it might still lead us to astronomical moral waste.” Does this clarify things?

  And what about more exotic ones? I argue they would also be greatly (though not entirely) reduced. Consider that a corrigible messiah may still hide poison for all of the humans at once, leading to an irrevocably terrible outcome. But why should it? If it thinks it is doing well by the humans, then its harmful actions ought to be transparent. Perhaps the AI system would’s actions would not be transparent if it intelligence was so radically great that it was inclined to act in fast an incomprehensible ways. But it is hard to see how we could know with confidence that such a radically intelligent AI is the kind we will soon be dealing with. And even if we are going to deal with that kind of AI, there could be other remedies that would be especially helpful in such scenarios. For example, an AI that permits informed oversight of its activities could be superb if it was already corrigible. Then, it could not only provide truthful explanations of its future plans but also respond to feedback on them. Overall, if we had an AI system that was (1) only a little bit superhumanly smart, (2) corrigible, and (3) providing informative explanations of its planned behaviour, then it would seem that we are in a pretty good spot.

  I agree with all this.

  I think you’re making an important point here, but here is how I would put it: If you have an AI system that is properly deferential to humans, you still need to rely on the humans not to give it any existentially catastrophic orders. But the corrigibility/​deferential behavior has changed the situation from one in which you’re relying on the metaphilosophical competence of the AI, to one in which you’re relying on the metaphilosphical competence on the human (albeit as filtered through the actions of the AI system). In the latter case, yes, you need to survive having a human’s power increased by some N-fold. (Not necessarily 10^15 as in the more extreme self-improvement scenarios, but by some N>1). So when you get a corrigible AI, you still need to be very careful with what you tell it to do, but your situation is substantially improved. Note that what I’m saying is at least in some tension with the traditional story of indirect normativity.

  I also agree with all this.

  Rather than trying to give the AI very general instructions for its interpretation, I’m saying that we should in the first instance try to stabilize the world so that we can do more metaphilosophical reasoning ourselves before trying to program an AI system that can carry out the conclusions of that thinking or perhaps continue it.

  I also agree with all this. I never imagined giving the corrigible AI extremely general instructions for its interpretation.

  Yes, an approval-directed agent might reward-hack by causing the human to approve of things that it does not value. And it might compromise the humans’ reasoning abilities while doing so. But why must the AI system’s metaphilosophical competence be the only defeator? Why couldn’t this be achieved by quantilizing, or otherwise throttling the agent’s capabilities? By restricting the agent’s activities to some narrow domain? By having the agent somehow be deeply uncertain about where the human’s approval mechanism resides? None of these seems clearly viable, but neither do any of them seem clearly impossible, especially in cases where the AI system’s capabilities are overall not far beyond those of its human operators.

  I think this captures my biggest update from your comment, and modified the ending of this post to reflect this. Throttling the AI’s power seems more viable than I’d previously thought, and seems like a pretty good way to significantly lower the risk of manipulation. That said, I think even extraordinary human persuadors might compromise human reasoning abilities, and I have fast takeoff intuitions that make it very hard for me to imagine an AGI that simultaneously

  • understands humans well enough to be corrigible

  • is superhumanly intelligent at engineering or global strategy

  • isn’t superhumanly capable of persuasion

  • wouldn’t corrupt humans (even if it tried to not corrupt them)

  I haven’t thought too much about this though, and this might just be a failure of my imagination.

  Overall, I’d say superintelligent messiahs are sometimes corrigible, and they’re more likely to be aligned if so.

  Agreed. Does the new title seem better? I was mostly trying to explicate a distinction between corrigibility and alignment, which was maybe obvious to you beforehand, and illustrate the astronomical waste that can result even if we avoid self-annihilation.

• zhukeepa 2 Apr 2018 20:21 UTC

  4 points

  in reply to: Wei Dai’s comment on: My take on agent foun­da­tions: for­mal­iz­ing metaphilo­soph­i­cal competence

  I agree I’ve been using “metaphilosophical competence” to refer to some combination of both rationality and philosophical competence. I have an implicit intuition that rationality, philosophical competence, and metaphilosophical competence all sort of blur into each other, such that being sufficient in any one of them makes you sufficient in all of them. I agree this is not obvious and probably confusing.

  To elaborate: sufficient metaphilosophical competence should imply broad philosophical competence, and since metaphilosophy is a kind of philosophy, sufficient philosophical competence should imply sufficient metaphilosophical competence. Sufficient philosophical competence would allow you to figure out what it means to act rationally, and cause you to act rationally.

  That rationality implies philosophical competence seems the least obvious. I suppose I think of philosophical competence as some combination of not being confused by words, and normal scientific competence—that is, given a bunch of data, figuring out which data is noise and which hypotheses fit the non-noisy data. Philosophy is just a special case where the data is our intuitions about what concepts should mean, the hypotheses are criteria/​definitions that capture these intuitions, and the datapoints happen to be extremely sparse and noisy. Some examples:

  • Section 1.1 in the logical induction paper lists a bunch of desiderata (“datapoints”) for what logical uncertainty is. The logical induction criterion is a criterion (“hypothesis”) that fits a majority of those datapoints.

  • The Von Neumann–Morgenstern utility theorem starts with a bunch of desiderata (“datapoints”) for rational behavior, and expected utility maximization is a criterion (“hypothesis”) that fits these datapoints.

  • I think both utilitarianism and deontology are moral theories (“hypotheses”) that fit a good chunk of our moral intuitions (“datapoints”). I also think both leave much to be desired.

  Philosophical progress seems objective and real like scientific progress—progress is made when a parsimonious new theory fits the data much better. One important way in which philosophical progress differs from scientific progress is that there’s much less consensus on what the data is or whether a theory fits it better, but I think this is mostly a function of most people being extremely philosophically confused, rather than e.g. philosophy being inherently subjective. (The “not being confused by words” component I identified mostly corresponds to the skill of identifying which datapoints we should consider in the first place, which of the datapoints are noise, and what it means for a theory to fit the data.)

  Relatedly, I think it is not a coincidence that the Sequences, which are primarily about rationality, also managed to deftly resolve a number of common philosophical confusions (e.g. MWI vs Copenhagen, free will, p-zombies).

  I also suspect that a sufficiently rational AGI would simply not get confused by philosophy the way humans do, and that it would feel to it from the inside like a variant of science. For example, it’s hard for me to imagine it tying itself up in knots trying to reason about theology. (I sometimes think about confusing philosophical arguments as adversarial examples for human reasoning...)

  Anyway, I agree this was all unclear and non-obvious (and plausibly wrong), and I’m happy to hear any suggestions for better descriptors. I literally went with “rationality” before “metaphilosophical competence”, but people complained that was overloaded and confusing...

  What links here?

  • Wei Dai's comment on Where to Draw the Boundaries? by Zack_M_Davis (16 Apr 2019 15:54 UTC; 18 points)
  • Wei Dai's comment on Can cor­rigi­bil­ity be learned safely? by Wei Dai (10 Apr 2018 7:42 UTC; 3 points)

• zhukeepa 2 Apr 2018 21:16 UTC

  4 points

  in reply to: corrigible-commenter’s comment on: Me­taphilo­soph­i­cal com­pe­tence can’t be dis­en­tan­gled from alignment

  I think values and decision-making processes can’t be disentangled, because I think people’s values often stem from their decision-making processes. For example, someone might be selfish because they perceive the whole world to be selfish and uncaring, and in return acts selfish and uncaring by default. This default behavior might cause the world to act selfishly and uncaringly toward him, further reinforcing his perception. If he fully understood this was happening (rather than the world just being fundamentally selfish), he might experiment with acting more generously with the rest of the world, and observe the rest of the world to act more generously in return, and in turn stop being selfish entirely.

  In general I expect amplification to improve decision-making processes substantially, but in most cases to not improve them enough. For example, it’s not clear that amplifying someone will cause them to observe that their own policy of selfishness is locking them into a fixed point that they could “Löb out of” into a more preferable fixed point. I expect this to be particularly unlikely if e.g. they believe their object-level values to be fixed and immutable, which migh result in a fairly pernicious epistemic pit.

  My intuition is that most decision-making processes have room for subtle but significant improvements, that most people won’t realize these improvements upon amplification, and that failing to make these improvements will result in catastrophic amounts of waste. As another example, it seems quite plausible to me that:

  • the vast majority of human-value-satisfaction (e.g. human flourishing, or general reduction of suffering) comes from acausally trading with distant superintelligences.

  • most people will never care about (or even realize) acausal trade, even upon amplification.

• zhukeepa 3 Apr 2018 2:59 UTC

  5 points

  in reply to: paulfchristiano’s comment on: Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah

  Do you think the AI-assisted humanity is in a worse situation than humanity is today? If we are metaphilosophically competent enough that we can make progress, why won’t we remain metaphilosphically competent enough once we have powerful AI assistants?

  Depends on who “we” is. If the first team that builds an AGI achieves a singleton, then I think the outcome is good if and only if the people on that team are metaphilosophically competent enough, and don’t have that competence corrupted by AI’s.

  In your hypothetical in particular, why do the people in the future—who have had radically more subjective time to consider this problem than we have, have apparently augmented their intelligence, and have exchanged massive amounts of knowledge with each other—make decisions so much worse than those that you or I would make today?

  If the team in the hypothetical is less metaphilosophically competent than we are, or have their metaphilosophical competence corrupted by the AI, then their decisions would turn out worse.

  I’m reminded of the lengthy discussion you had with Wei Dai back in the day. I share his picture of which scenarios will get us something close to optimal, his beliefs that philosophical ignorance might persist indefinitely, his skepticism about the robustness of human reflection, and his skepticism that human values will robustly converge upon reflection.

  Is that an accurate characterization?

  I would say so. Another fairly significant component is my model that humanity makes updates by having enough powerful people paying enough attention to reasonable people, enough other powerful people paying attention to those powerful people, and with everyone else roughly copying the beliefs of the powerful people. So, good memes --> reasonable people --> some powerful people --> other powerful people --> everyone else.

  AI would make some group of people far more powerful than the rest, which screws up the chain if that group don’t pay much attention to reasonable people. In that case, they (and the world) might just never become reasonable. I think this would happen if ISIS took control, for example.

  Other than disagreeing, my main complaint is that this doesn’t seem to have much to do with AI. Couldn’t you tell exactly the same story about human civilization proceeding along its normal development trajectory, never building an AI, but gradually uncovering new technologies and becoming smarter?

  I would indeed expect this by default, particularly if one group with one ideology attains decisive control over the world. But if we somehow manage to avoid that (which seems unlikely to me, given the nature of technological progress), I feel much more optimistic about metaphilosophy continuing to progress and propagate throughout humanity relatively quickly.

• zhukeepa 3 Apr 2018 3:38 UTC

  2 points

  in reply to: paulfchristiano’s comment on: Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah

  I don’t think this is right. The agent is optimized to choose actions which, when shown to a human, receive high approval. It’s not optimized to pick actions which, when executed, cause the agent to receive high approval in the future.

  I think optimizing for high approval now leaves a huge number of variables unconstrained. For example, I could absolutely imagine a corrigible AI with ISIS suicide bomber values that consistently receives high approval from its operator and eventually turns its operator into an ISIS suicide bomber. (Maybe not all operators, but definitely some operators.)

  Given the constraint of optimizing for high approval now, in what other directions would our corrigible AI try to optimize? Some natural guesses would be optimizing for future approval, or optimizing for its model of its operator’s extrapolated values (which I would distrust unless I had good reason to trust its extrapolation process). If it were doing either, I’d be very scared about getting corrupted. But you’re right that it may not optimize for us turning into yes-men in particular.

  I suspect this disagreement is related to our disagreement about the robustness of human reflection. Actually, the robustness of human reflection is a crux for me—if I thought human reflection were robust, then I think an AI that continously optimizes for high approval now leaves few important variables unconstrained, and would lead us to very good outcomes by default. Is this a crux for you too?

  What links here?

  • Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah by zhukeepa (1 Apr 2018 6:20 UTC; 28 points)
  • zhukeepa's comment on Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah by zhukeepa (3 Apr 2018 5:11 UTC; 1 point)

• zhukeepa 3 Apr 2018 3:57 UTC

  1 point

  in reply to: paulfchristiano’s comment on: Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah

  If the AI is optimizing its behavior to have some effect on the human, then that’s practically the central case the concept of corrigibility is intended to exclude. I don’t think it matters what the AI thinks about what the AI is doing, it just matters what optimization power it is applying.

  See my comment re: optimizing for high approval now vs. high approval later.

  If you buy (as I do) that optimizing for high approval now leaves a huge number of important variables unconstrained, I don’t see how it make sense to talk about an AI optimizing for high approval now without also optimizing to have some effect on the human, because the unconstrained variables are about effects on the human. If there were a human infant in the wilderness and you told me to optimize for keeping it alive without optimizing for any other effects on the infant, and you told me I’d be screwing up if I did optimize for other effects, I would be extremely confused about how to proceed.

  If you don’t buy that optimizing for high approval now leaves a huge number of important variables unconstrained, then I agree that the AI optimizing its behavior to have some effects on the human should be ruled out by the definition of corrigibility.

  What links here?

  • Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah by zhukeepa (1 Apr 2018 6:20 UTC; 28 points)
  • zhukeepa's comment on Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah by zhukeepa (3 Apr 2018 5:11 UTC; 1 point)

• zhukeepa 3 Apr 2018 5:11 UTC

  1 point

  in reply to: RyanCarey’s comment on: Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah

  Thanks Ryan!

  What I see to be the main message of the article as currently written is that humans controlling a very powerful tool (especially AI) could drive themselves into a suboptimal fixed point due to insufficient philosophical sophistication.

  This I agree with.

  Hurrah!

  At this round of edits, my main objection would be to the remark that the AI wants us to act as yes-men, which seems dubious if the agent is (i) an Act-based agent or (ii) sufficiently broadly uncertain over values.

  I no longer think it wants us to turn into yes-men, and edited my post accordingly. I still think it will be incentivized to corrupt us, and I don’t see how being an act-based agent would be sufficient, though it’s likely I’m missing something. I agree that if it’s sufficiently broadly uncertain over values then we’re likely to be fine, but in my head that unpacks into “if we knew the AI were metaphilosophically competent enough, we’d be fine”, which doesn’t help things much.

• zhukeepa 5 Apr 2018 5:04 UTC

  LW: 12 AF: 4

  AF

  in reply to: Wei Dai’s comment on: Can cor­rigi­bil­ity be learned safely?

  I thought more about my own uncertainty about corrigibility, and I’ve fleshed out some intuitions on it. I’m intentionally keeping this a high-level sketch, because this whole framing might not make sense, and even if it does, I only want to expound on the portions that seem most objectionable.

  Suppose we have an agent A optimizing for some values V. I’ll call an AI system S high-impact calibrated with respect to A if, when A would consider an action “high-impact” with respect to V, S will correctly classify it as high-impact with probability at least 1-ɛ, for some small ɛ.

  My intuitions about corrigibility are as follows:

  1. If you’re not calibrated about high-impact, catastrophic errors can occur. (These are basically black swans, and black swans can be extremely bad.)

  2. Corrigibility depends critically on high-impact calibration (when your AI is considering doing a high-impact thing, it’s critical that it knows to check that action with you).

  3. To learn how to be high-impact calibrated w.r.t. A, you will have to generalize properly from training examples of low/​high-impact (i.e. be robust to distributional shift).

  4. To robustly generalize, you’re going to need the ontologies /​ internal representations that A is using. (In slightly weirder terms, you’re going to have to share A’s tastes/​aesthetic.)

  5. You will not be able to learn those ontologies unless you know how to optimize for V the way A is optimizing for V. (This is the core thing missing from the well-intentioned extremley non-neurotypical assistant I illlustrated.)

  6. If S’s “brain” starts out very differently from A’s “brain”, S will not be able to model A’s representations unless S is significantly smarter than A.

  In light of this, for any agent A, some value V they’re optimizing for, and some system S that’s assisting A, we can ask two important questions:

  (I) How well can S learn A’s representations?

  (II) If the representation is imperfect, how catastrophic might the resulting mistakes be?

  In the case of a programmer (A) building a web app trying to make users happy (V), it’s plausible that some run-of-the-mill AI system (S) would learn a lot of the important representations right and a lot of the important representations wrong, but it also seems like none of the mistakes are particularly catastrophic (worst case, the programmer just reverts the codebase.)

  In the case of a human (A) trying to make his company succeed (V), looking for a new CEO (S) to replace himself, it’s usually the case that the new CEO doesn’t have the same internal representations as the founder. If they’re too different, the result is commonly catastrophic (e.g. if the new CEO is an MBA with “more business experience”, but with no vision and irreconcilable taste). Some examples:

  • For those who’ve watched HBO’s Silicon Valley, Action Jack Barker epitomizes this.

  • When Sequoia Capital asked Larry and Sergey to find a new CEO for Google, they hemmed and hawed until they found one who had a CS Ph.D and went to Burning Man, just like they did. (Fact-check me on this one?)

  • When Apple ousted Steve Jobs, the company tanked, and only after he was hired back as CEO did the company turn around and become the most valuable company in the world.

  (It’s worth noting that if the MBA got hired as a “faux-CEO”, where the founder could veto any of the MBA’s proposals, the founders might make some use of him. But the way in which he’d be useful is that he’d effectively be hired for some non-CEO position. In this picture, the founders are still doing most of the cognitive work in running the company, while the MBA ends up relegated to being a “narrow tool intelligence utilized for boring business-y things”. It’s also worth noting that companies care significantly about culture fit when looking for people to fill even mundane MBA-like positions...)

  In the case of a human (A) generically trying to optimize for his values (V), with an AGI trained to be corrigible (S) assisting, it seems quite unlikely that S would be able to learn A’s relevant internal representations (unless it’s far smarter and thus untrustworthy), which would lead to incorrect generalizations. My intuition is that if S is not much smarter than A, but helping in extremely general ways and given significant autonomy, the resulting outcome will be very bad. I definitely think this if S is a sovereign, but also think this if e.g. it’s doing a thousand years’ worth of human cognitive work in determining if a newly distilled agent is corrigible, which I think happens in ALBA. (Please correct me if I botched some details.)

  Paul: Is your picture that the corrigible AI learns the relevant internal representations in lockstep with getting smarter, such that it manages to hit a “sweet spot” where it groks human values but isn’t vastly superintelligent? Or do you think it doesn’t learn the relevant internal representations, but its action space is limited enough that none of its plausible mistakes would be catastrophic? Or do you think one of my initial intuitions (1-6) is importantly wrong? Or do you think something else?

  Two final thoughts:

  • The way I’ve been thinking about corrigibility, there is a simple core to corrigibility, but it only applies when the subagent can accurately predict any judgment you’d make of the world, and isn’t much more powerful than you. This is the case if e.g. the subagent starts as a clone of you, and is not the case if you’re training it from scratch (because it’ll either be too dumb to understand you, or too smart to be trustworthy). I’m currently chewing on some ideas for operationalizing this take on corrigibility using decision theory.

  • None of this analysis takes into account that human notions of “high-impact” are often wrong. Typical human reasoning processes are pretty susceptible to black swans, as history shows. (Daemons sprouting would be a subcase of this, where naive human judgments might judge massive algorithmic searches to be low-impact.)

  What links here?

  • Chi Nguyen's comment on My Un­der­stand­ing of Paul Chris­ti­ano’s Iter­ated Am­plifi­ca­tion AI Safety Re­search Agenda by Chi Nguyen (17 Aug 2020 22:49 UTC; 2 points)

• zhukeepa 5 Apr 2018 6:22 UTC

  1 point

  in reply to: paulfchristiano’s comment on: Cor­rigible but mis­al­igned: a su­per­in­tel­li­gent messiah

  Oops, I do think that’s what I meant. To explain my wording: when I imagined a “system optimizing for X”, I didn’t imagine that system trying its hardest to do X, I imagined “a system for which the variable Z it can best be described as optimizing is X”.

  To say it all concretely another way, I mean that there are a bunch of different systems $S_1,S_2,...,S_n$ that, when “trying to optimize for X as hard as possible” all look to us like they optimize for X successfully, but do so via methods $M_1,M_2,...,M_n$ that lead to vastly different (and generally undesirable) endstates $Y_1,Y_2,..,Y_n$ like the one described in this post, or one where the operators become ISIS suicide bombers. In this light, it seem more accurate to describe $S_i$ as optimizing for $Y_i$ instead of X, even though $S_i$ is trying to optimize for X and optimizes it pretty successfully. But I really don’t want a superintelligent system optimizing for some Y that is not my values.

  As a possibly related general intuition, I think the space of outcomes that can result from having a human follow a sequence of suggestions, each of which they’d enthusiastically endorse, is massive, and that most of these outcomes are undesirable. (It’s possible that one crisp articulation of “sufficient metaphilosophical competence” is that following a sequence of suggestions, each of which you’d enthusiastically endorse, is actually good for you.)

  On reflection, I agree that neither future approval nor idealized preferences are particularly likely, and that whatever Y is would actually look very alien.

  What links here?

  • zhukeepa's comment on Challenges to Chris­ti­ano’s ca­pa­bil­ity am­plifi­ca­tion proposal by Eliezer Yudkowsky (20 May 2018 19:26 UTC; 11 points)

• zhukeepa 8 Apr 2018 18:08 UTC

  LW: 7 AF: 3

  AF

  in reply to: paulfchristiano’s comment on: Can cor­rigi­bil­ity be learned safely?

  I really like that list of points! Not that I’m Rob, but I’d mentally classified each of those as alignment failures, and the concern I was trying to articulate was that, by default, I’d expect an AI trying to do the right thing will make something like one of these mistakes. Those are good examples of the sorts of things I’d be scared of if I had a well-intentioned non-neurotypical assistant. Those are also what I was referring to when I talked about “black swans” popping up. And when I said:

  2. Corrigibility depends critically on high-impact calibration (when your AI is considering doing a high-impact thing, it’s critical that it knows to check that action with you).

  I meant that, if an AI trying to do the right thing was considering one of these actions, for it to be safe it should consult you before going ahead with any one of these. (I didn’t mean “the AI is incorrigible if it’s not high-impact calibrated”, I meant “the AI, even if corrigible, would be unsafe it’s not high-impact calibrated”.)

  If these kinds of errors are included in “alignment,” then I’d want some different term that referred to the particular problem of building AI that was trying to do the right thing, without including all of the difficulty of figuring out what is right (except insofar as “figure out more about what is right” is one way to try to build an AI that is trying to do the right thing.)

  I think I understand your position much better now. The way I’ve been describing “ability to figure out what is right” is “metaphilosophical competence”, and I currently take the stance that an AI trying to do the right thing will by default be catastrophic if it’s not good enough at figuring out what is right, even if it’s corrigible.

  What links here?

  • zhukeepa's comment on Can cor­rigi­bil­ity be learned safely? by Wei Dai (8 Apr 2018 19:57 UTC; 7 points)

• zhukeepa 8 Apr 2018 19:57 UTC

  7 points

  in reply to: paulfchristiano’s comment on: Can cor­rigi­bil­ity be learned safely?

  I replied about (2) and black swans in a comment way down.

  I’m curious to hear more about your thoughts about (4).

  To flesh out my intuitions around (4) and (5): I think there are many tasks where a high-dimensional and difficult to articulate piece of knowledge is critical for completing the task. For example:

  • if you’re Larry or Sergey trying to hire a new CEO, you need your new CEO to be a culture fit. Which in this case means something like “being technical, brilliant, and also a hippie at heart”. It’s really, really hard to communicate this to a slick MBA. Especially the “be a hippie at heart” part. Maybe if you sent them to Burning Man and had them take a few drugs, they’d grok it?

  • if you’re Bill Gates hiring a new CEO, you should make sure your new CEO is also a developer at heart, not a salesman. Otherwise, you might hire Steve Ballmer, who drove Microsoft’s revenues through the roof for a few years, but also had little understanding of developers (for example he produced an event where he celebrated developers in a way developers don’t tend to like being celebrated). This led to an overall trend of the company losing its technical edge, and thus its competitive edge… this was all while Ballmer had worked with Gates at Microsoft for two decades. If Ballmer was a developer, he may have been able avoid this, but he very much wasn’t.

  • if you’re a self-driving car engineer delegating image classification to a modern-day neural net, you’d really want its understanding of what the classifications mean to match yours, lest they be susceptible to clever adversarial attacks. Humans understand the images to represent projections of crisp three-dimensional objects that exist in a physical world; image classifiers don’t, which is why they can get fooled so easily by overlays of random patterns. Maybe it’s possible to replicate this understanding without being an embodied agent, but it seems you’d need something beyond training a big neural net on a large collection of images, and making incremental fixes.

  • if you’re a startup trying to build a product, it’s very hard to do so correctly if you don’t have a detailed implicit model of your users’ workflows and pain points. It helps a lot to talk to them, but even then, you may only be getting 10% of the picture if you don’t know what it’s like to be them. Most startups die by not having this picture, flying blind, and failing to acquire any users.

  • if you’re trying to help your extremely awkward and non-neurotypical friend find a romantic partner, you might find it difficult to convey what exactly is so bad about carrying around slips of paper with clever replies, and pulling them out and reading from them when your date says something you don’t have a reply to. (It’s not that hard to convey why doing this particuar thing is bad. It’s hard to convey what exactly about it is bad, that would have him properly generalize and avoid all classes of mistakes like this going forward, rather than just going like “Oh, pulling out slips of paper is jarring and might make her feel bad, so I’ll stop doing this particular thing”.) (No, I did not make up this example.)

  In these sorts of situations, I wouldn’t trust an AI to capture my knowledge/​understanding. It’s often tacit and perceptual, it’s often acquired through being a human making direct contact with reality, and it might require a human cognitive architecture to even comprehend in the first place. (Hence my claims that proper generalization requires having the same ontologies as the overseer, which they obtained from their particular methods of solving a problem.)

  In general, I feel really sketched about amplifying oversight, if the mechanism involves filtering your judgment through a bunch of well-intentioned non-neurotypical assistants, since I’d expect the tacit understandings that go into your judgment to get significantly distorted. (Hence my curiosity about whether you think we can avoid the judgment getting significantly distorted, and/​or whether you think we can do fine even with significantly distorted judgment.)

  It’s also pretty plausible that I’m talking completely past you here; please let me know if this is the case.