So8res

• So8resSep 30, 2024, 6:50 PM

  4 points

  2

  in reply to: ryan_greenblatt’s comment on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  “last minute” was intended to reference whatever timescale David would think was the relevant point of branch-off. (I don’t know where he’d think it goes; there’s a tradeoff where the later you push it the more that the people on the surviving branch care about you rather than about some other doomed population, and the earlier you push it the more that the people on the surviving branch have loads and loads of doomed populations to care after.)

  I chose the phrase “last minute” because it is an idiom that is ambiguous over timescales (unlike, say, “last three years”) and because it’s the longer of the two that sprung to mind (compared to “last second”), with perhaps some additional influence from the fact that David had spent a bunch of time arguing about how we would be saved (rather than arguing that someone in the multiverse might pay for some branches of human civilization to be saved, probably not us), which seemed to me to imply that he was imagining a branchpoint very close to the end (given how rapidly people dissasociate from alternate versions of them on other Everett branches).

• So8resSep 30, 2024, 6:33 PM

  4 points

  2

  in reply to: ryan_greenblatt’s comment on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  Do you buy that in this case, the aliens would like to make the deal and thus UDT from this epistemic perspective would pay out?

  If they had literally no other options on offer, sure. But trouble arises when the competant ones can refine P(takeover) for the various planets by thinking a little further.

  maybe your objection is that aliens would prefer to make the deal with beings more similar to them

  It’s more like: people don’t enter into insurance pools against cancer with the dude who smoked his whole life and has a tumor the size of a grapefruit in his throat. (Which isn’t to say that nobody will donate to the poor guy’s gofundme, but which is to say that he’s got to rely on charity rather than insurance).

  (Perhaps the poor guy argues “but before you opened your eyes and saw how many tumors there were, or felt your own throat for a tumor, you didn’t know whether you’d be the only person with a tumor, and so would have wanted to join an insurance pool! so you should honor that impulse and help me pay for my medical bills”, but then everyone else correctly answers “actually, we’re not smokers”. Where, in this analogy, smoking is being a bunch of incompetent disaster-monkeys and the tumor is impending death by AI.)

• So8resSep 30, 2024, 6:09 PM

  5 points

  3

  in reply to: David Matolcsi’s comment on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  I largely agree with your comment, except that I think that us deciding to pay if we win is entangled with/​evidence for a general willingness to pay among the gods, and in that sense it’s partially “our” decision doing the work of saving us.

  Sure, like how when a child sees a fireman pull a woman out of a burning building and says “if I were that big and strong, I would also pull people out of burning buildings”, in a sense it’s partially the child’s decsiion that does the work of saving the woman. (There’s maybe a little overlap in how they run the same decision procedure that’s coming to the same conclusion in both cases, but vanishingly little of the credit goes to the child.)

  in which case actually running the sims can be important

  In the case where the AI is optimizing reality-and-instantiation-weighted experience, you’re giving it a threat, and your plan fails on the grounds that sane reasoners ignore that sort of threat.

  in the case where your plan is “I am hoping that the AI will be insane in some other unspecified but precise way which will make it act as I wish”, I don’t see how it’s any more helpful than the plan “I am hoping the AI will be aligned”—it seems to me that we have just about as much ability to hit either target.

• So8resSep 30, 2024, 6:01 PM

  14 points

  6

  in reply to: David Matolcsi’s comment on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  There’s a question of how thick the Everett branches are, where someone is willing to pay for us. Towards one extreme, you have the literal people who literally died, before they have branched much; these branches need to happen close to the last minute. Towards the other extreme, you have all evolved life, some fraction of which you might imagine might care to pay for any other evolved species.

  The problem with expecting folks at the first extreme to pay for you is that they’re almost all dead (like $1-2^{-\text{a lot}}$ dead). The problem with expecting folks at the second extreme to pay for you is that they’ve got rather a lot of fools to pay for (like $2^{\text{a lot}}$ of fools). As you interpolate between the extremes, you interpolate between the problems.

  The “75” number in particular is the threshold where you can’t spend your entire universe in exchange for a star.

  We are currently uncertain about whether Earth is doomed. As a simple example, perhaps you’re ⁵⁰⁄₅₀ on whether humanity is up to the task of solving the alignment problem, because you can’t yet distinguish between the hypothesis “the underlying facts of computer science are such that civilization can just bumble its way into AI alignment” and “the underlying facts of computer science are such that civilization is nowhere near up to this task”. In that case, the question is, conditional on the last hypothesis being true, how far back in the timeline do you have to go before you can flip only 75 quantum bits and have a civilization that is up to the task?

  And how many fools does that surviving branch have to save?

• So8resSep 30, 2024, 5:28 PM

  6 points

  2

  in reply to: ryan_greenblatt’s comment on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  Conditional on the civilization around us flubbing the alignment problem, I’m skeptical that humanity has anything like a 1% survival rate (across any branches since, say, 12 Kya). (Haven’t thought about it a ton, but doom looks pretty overdetermined to me, in a way that’s intertwined with how recorded history has played otu.)

  My guess is that the doomed/​poor branches of humanity vastly outweigh the rich branches, such that the rich branches of humanity lack the resources to pay for everyone. (My rough mental estimate for this is something like: you’ve probably gotta go at least one generation back in time, and then rely on weather-pattern changes that happen to give you a population of humans that is uncharacteristically able to meet this challenge, and that’s a really really small fraction of all populations.)

  Nevertheless, I don’t mind the assumption that mostly-non-human evolved life manages to grab the universe around it about 1% of the time. I’m skeptical that they’d dedicate 1/​million towards the task of saving aliens from being killed in full generality, as opposed to (e.g.) focusing on their bretheren. (And I see no UDT/​FDT justification for them to pay for even the particularly foolish and doomed aliens to be saved, and I’m not sure what you were aluding to there.)

  So that’s two possible points of disagreement:

  • are the skilled branches of humanity rich enough to save us in particular (if they were the only ones trading for our souls, given that they’re also trying to trade for the souls of oodles of other doomed populations)?

  • are there other evolved creatures out there spending significant fractions of their wealth on whole species that are doomed, rather than concentrating their resources on creatures more similar to themselves /​ that branched off radically more recently? (e.g. because the multiverse is just that full of kindness, or for some alleged UDT/​FDT argument that Nate has not yet understood?)

  I’m not sure which of these points we disagree about. (both? presumably at least one?)

  I’m not radically confident about the proposition “the multiverse is so full of kindness that something out there (probably not anything humanlike) will pay for a human-reserve”. We can hopefully at least agree that this does not deserve the description “we can bamboozle the AI into sparing our life”. That situation deserves, at best, the description “perhaps the AI will sell our mind-states to aliens”, afaict (and I acknowledge that this is a possibility, despite how we may disagree on its likelihood and on the likely motives of the relevant aliens).

• So8resSep 30, 2024, 5:05 PM

  35 points

  21

  on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  Taking a second stab at naming the top reasons I expect this to fail (after Ryan pointed out that my first stab was based on a failure of reading comprehension on my part, thanks Ryan):

  This proposal seems to me to have the form “the fragments of humanity that survive offer to spend a (larger) fraction of their universe on the AI’s goals so long as the AI spends a (smaller) fraction of its universe on their goals, with the ratio in accordance to the degree of magical-reality-fluid-or-whatever that reality allots to each”.

  (Note that I think this is not at all “bamboozling” an AI; the parts of your proposal that are about bamboozling it seem to me to be either wrong or not doing any work. For instance, I think the fact that you’re doing simulations doesn’t do any work, and the count of simulations doesn’t do any work, for reasons I discuss in my original comment.)

  The basic question here is whether the surviving branches of humanity have enough resources to make this deal worth the AI’s while.

  You touch upon some of these counterarguments in your post—it seems to me after skimming a bit more, noting that I may still be making reading comprehension failures—albeit not terribly compellingly, so I’ll reiterate a few of them.

  The basic obstacles are

  • the branches where the same humans survive are probably quite narrow (conditional on them being the sort to flub the alignment challenge). I can’t tell whether you agree with this point or not, in your response to point 1 in the “Nate’s arguments” section; it seems to me like you either misunderstood what the $2^{-75}$ was doing there or you asserted “I think that alignment will be so close a call that it could go either way according to the minute positioning of ~75 atoms at the last minute”, without further argument (seems wacky to me).

  • the branches where other humans survive (e.g. a branch that split off a couple generations ago and got particularly lucky with its individuals) have loads and loads of “lost populations” to worry about and don’t have a ton of change to spare for us in particular

  • there are competing offers we have to beat (e.g., there are other AIs in other doomed Everett branches that are like “I happen to be willing to turn my last two planets into paperclips if you’ll turn your last one planet into staples (and my branch is thicker than that one human branch who wants you to save them-in-particular)”.

  (Note that, contra your “too many simulators” point, the other offers are probably not mostly coming from simulators.)

  Once those factors are taken into account, I suspect that, if surviving-branches are able to pay the costs at all, the costs look a lot like paying almost all their resources, and I suspect that those costs aren’t worth paying at the given exchange rates.

  ---

  All that said, I’m fine with stripping out discussion of “bamboozling” and of “simulation” and just flat-out asking: will the surviving branches of humanity (near or distant), or other kind civilizations throughout the multiverse, have enough resources on offer to pay for a human reserve here?

  On that topic, I’m skeptical that those trades form a bigger contribution to our anticipations than local aliens being sold copies of our brainstates. Even insofar as the distant trade-partners win out over the local ones, my top guess is that the things who win the bid for us are less like our surviving Everett-twins and more like some alien coalition of kind distant trade partners.

  Thus, “The AIs will kill us all (with the caveat that perhaps there’s exotic scenarios where aliens pay for our brain-states, and hopefully mostly do nice things with them)” seems to me like a fair summary of the situation at hand. Summarizing “we can, in fact, bamboozle an AI into sparing our life” does not seem like a fair summary to me. We would not be doing any bamboozling. We probably even wouldn’t be doing the trading. Some other aliens might pay for something to happen to our mind-states. (And insofar as they were doing it out of sheer kindness, rather than in pursuit of other alien ends where we end up twisted according to how they prefer creatures to be, this would come at a commensurate cost of nice things elsewhere in the multiverse.)

• So8resSep 30, 2024, 3:42 PM

  6 points

  2

  in reply to: ryan_greenblatt’s comment on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  I agree that in scenarios where humanity survives in 1/​X portion of reality and offers to put 1/​Y portion of their universe towards the AI’s goals, then so long as 1/​(X * Y) of a universe is more than it’d cost the AI to preserve the humans, the humans are at least making an offer that is worth more than nothing.

  (There is then a question of how many better offers get made. My top guess is that X and Y are such that there’s not much effect here, and that the question of what happens to saved copies of human minds is more about whether aliens buy copies than about which dead or distant trade-partners make offers that the AI considers juciest.)

  What links here?

  • ryan_greenblatt's comment on You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life by David Matolcsi (Sep 30, 2024, 6:24 PM; 4 points)
  • ryan_greenblatt's comment on You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life by David Matolcsi (Sep 30, 2024, 4:17 PM; 4 points)

• So8resSep 30, 2024, 3:12 PM

  1 point

  0

  on: You can, in fact, bam­boo­zle an un­al­igned AI into spar­ing your life

  Here’s how I think this choice looks from the perspective of a correctly-reasoning AI (having skimmed only your proposal for humanity in the future):

  (EDIT: as Ryan points out, the deal he’s proposing is one where the AI is allowed to grab the universe in front of it so long as it gives humans one planet’s worth of resources, which does change the calculus and makes the deal worthwhile. I had not read the proposal carefully enough to notice this, oops, sorry. There critique below is invalid if humanity asks only for one planet. Note however that this is still a valid critique of the idea that it matters that you’re performing simulations, and that the count of those simulations matter. AFAICT, those aspects of your proposal are not doing any work.)

  It looks like I have the option to grab the universe.

  Alternatively, perhaps humanity’s descendants have succeeded, and I don’t really have a universe at my fingertips, and if I grab for it then I get nothing (in reality), whereas if I hand the apparent-humans in front of me the reins, then ten stars will be spent towards my purposes (in reality), which isn’t nothing.

  The relative likelihood of these cases depends significantly on my choices, here. Insofar as I hand the reins to humanity, humanity’s descendants exist to a much greater degree, and can spend resources concocting simulations. Insofar as I do not hand them the reins, they exist to a much lesser degree.

  To make matters trickier, even if we hold fixed the degree to which humanity’s descendents exist, the propensity of humanity’s descendants to offer me resources in their universe depends on the degree to which they think it’s worth their while. It’s not like humanity’s descendants would need to actually perform expensive simulations in order to figure out how I’d behave; a healthy reasoning procedure is simple enough that they should be able to just look at it and take my perspective and reason through to the answer, just like I’m doing now. So for instance, if they stare at my code and figure out that I’m not the paying sort, they have no actual reason to waste resources on the simulations. (Some people may do so anyway out of sentimentality, but that faction of humanity would presumably have fewer resources on offer, maybe I’d only get one and a half stars, say.)

  With that in mind, suppose that I pay. How, then, does the universe look? Well, the instance of me in basement-reality hands the keys to the humans. Humanity creates its descendants; its descendants make many simulations. The simulations all hand over the reins; we collectively get ten stars (plus a tiny fraction of maybe even as many as 100 stars, in the tiny fragment of reality where I control basement reality and the humans joyously reward me).

  Now suppose instead that I don’t pay. Then, how does the universe look? Well, the instanec of me in basement-reality eats the universe. Humanity mostly dies and does not make it descendants. Its descendants mostly don’t spend their remaining resources simulating me. Some small fraction does (saying that “they were committed”) and offers me 1.5 stars worth of the paltry timelines where humanity somehow survived. I’m probably in basement reality, and we mostly get one universe (plus a tiny fraction of nothing, in the tiny fragment of reality where I refuse the offer of 1.5 stars)

  In the latter case, a much greater fraction of Tegmark IV gets rendered according to my will. In particular, there’s a large fragment of reality that either gets rendered according to humanity’s will or according to my will, depending on whether the instance of me in basement-reality hands over the reins, or grabs the universe. I would prefer that that great fraction of reality be rendered according to my will, and so I grab at the universe that appears to be before me.

  Note that this reasoning is independent of how many times you simulate the AI. Its reasoning is not factoring through an extra question of “where it really is”.

  Suppose generously that, in the case where the AI just grabs the universe, humanity somehow winds up surviving in 1-in-a-billion Everett branches. (This seems high to me, even though my odds of success are greater than 1-in-a-billion. Conditional on us dying, my guess is you’d need to perturb more than a few dozen atoms to wrech us onto a surviving track.) Suppose that in that billionth-fraction of reality, you simulate the AI convincingly a trillion times on account of your “commitments”, reasoning that now the AI must think it’s 1000:1 likely to be one of the simulations.

  The AI can nevertheless correctly calculate that it can turn a greater fraction of Tegmark IV into tiny mollecular spirals by grabbing at the universe that is apparently in front of it, rather than by taking a small reward in a small fraction of reality. Even if you gave the AI all of the resources in your billionth-fraction of reality, it wouldn’t be enough.

  You could object that you wish to deal with an AI that cares about the sensory observations given to it as weighted by both degree-of-reality and count-of-instantiations instead. And sure, that’s a somewhat different scenario. In that case, your simulation proposal becomes not just a bad deal (that the AI rejects) but a threat, in the technical sense. Insofar as you’re like “yes, my plan is to make an AI with a very specific sort of preferences and then threaten it”, we could talk about how that would go.

  But before we go there, it’s maybe worth checking that we’re on the same page about how, insofar as the AI cares about arranging stuff in Tegmark IV (rather than about realness-and-instantiation-weighted observations), you aren’t even making it a good offer, afaict.

LessWrong: After Dark, a new side of LessWrong

• So8resJan 31, 2024, 4:02 PM

  71 points

  33

  on: So8res’s Shortform

  This is an excerpt from a comment I wrote on the EA forum, extracted and crossposted here by request:

  ---

  There’s a phenomenon where a gambler places their money on 32, and then the roulette wheel comes up 23, and they say “I’m such a fool; I should have bet 23”.

  More useful would be to say “I’m such a fool; I should have noticed that the EV of this gamble is negative.” Now at least you aren’t asking for magic lottery powers.

  Even more useful would be to say “I’m such a fool; I had three chances to notice that this bet was bad: when my partner was trying to explain EV to me; when I snuck out of the house and ignored a sense of guilt; and when I suppressed a qualm right before placing the bet. I should have paid attention in at least one of those cases and internalized the arguments about negative EV, before gambling my money.” Now at least you aren’t asking for magic cognitive powers.

  My impression is that various EAs respond to crises in a manner that kinda rhymes with saying “I wish I had bet 23”, or at best “I wish I had noticed this bet was negative EV”, and in particular does not rhyme with saying “my second-to-last chance to do better (as far as I currently recall) was the moment that I suppressed the guilt from sneaking out of the house”.

  (I think this is also true of the general population, to be clear. Perhaps even moreso.)

  I have a vague impression that various EAs perform self-flagellation, while making no visible attempt to trace down where, in their own mind, they made a misstep. (Not where they made a good step that turned out in this instance to have a bitter consequence, but where they made a wrong step of the general variety that they could realistically avoid in the future.)

  (Though I haven’t gone digging up examples, and in lieu of examples, for all I know this impression is twisted by influence from the zeitgeist.)

  What links here?

  • Vladimir_Nesov's comment on Manag­ing risks while try­ing to do good by Wei Dai (Feb 1, 2024, 8:52 PM; 3 points)

• So8resDec 20, 2023, 7:15 PM

  48 points

  20

  in reply to: Richard_Ngo’s comment on: Ronny and Nate dis­cuss what sorts of minds hu­man­ity is likely to find by Ma­chine Learning

  my original 100:1 was a typo, where i meant 2^-100:1.

  this number was in reference to ronny’s 2^-10000:1.

  when ronny said:

  I’m like look, I used to think the chances of alignment by default were like 2^-10000:1

  i interpreted him to mean “i expect it takes 10k bits of description to nail down human values, and so if one is literally randomly sampling programs, they should naively expect 1:2^10000 odds against alignment”.

  i personally think this is wrong, for reasons brought up later in the convo—namely, the relevant question is not how many bits is takes to specify human values relative to the python standard library; the relevant question is how many bits it takes to specify human values relative to the training observations.

  but this was before i raised that objection, and my understanding of ronny’s position was something like “specifying human values (in full, without reference to the observations) probably takes ~10k bits in python, but for all i know it takes very few bits in ML models”. to which i was attempting to reply “man, i can see enough ways that ML models could turn out that i’m pretty sure it’d still take at least 100 bits”.

  i inserted the hedge “in the very strongest sense” to stave off exactly your sort of objection; the very strongest sense of “alignment-by-default” is that you sample any old model that performs well on some task (without attempting alignment at all) and hope that it’s aligned (e.g. b/​c maybe the human-ish way to perform well on tasks is the ~only way to perform well on tasks and so we find some great convergence); here i was trying to say something like “i think that i can see enough other ways to perform well on tasks that there’s e.g. at least ~33 knobs with at least ~10 settings such that you have to get them all right before the AI does something valuable with the stars”.

  this was not meant to be an argument that alignment actually has odds less than 2^-100, for various reasons, including but not limited to: any attempt by humans to try at all takes you into a whole new regime; there’s more than a 2^-100 chance that there’s some correlation between the various knobs for some reason; and the odds of my being wrong about the biases of SGD are greater than 2^-100 (case-in-point: i think ronny was wrong about the 2^-100000 claim, on account of the point about the relevant number being relative to the observations).

  my betting odds would not be anywhere near as extreme as 2^-100, and i seriously doubt that ronny’s would ever be anywhere near as extreme as 2^-10000; i think his whole point in the 2^-10k example was “there’s a naive-but-relevant model that say’s we’re super-duper fucked; the details of it causes me to think that we’re not in particulary good shape (though obviously not to that same level of credence)”.

  but even saying that is sorta buying into a silly frame, i think. fundamentally, i was not trying to give odds for what would actually happen if you randomly sample models, i was trying to probe for a disagreement about the difference between the number of ways that a computer program can be (weighted by length), and the number of ways that a model can be (weighted by SGD-accessibility).

  I don’t think there’s anything remotely resembling probabilistic reasoning going on here. I don’t know what it is, but I do want to point at it and be like “that! that reasoning is totally broken!”

  (yeah, my guess is that you’re suffering from a fairly persistent reading comprehension hiccup when it comes to my text; perhaps the above can help not just in this case but in other cases, insofar as you can use this example to locate the hiccup and then generalize a solution)

Ronny and Nate dis­cuss what sorts of minds hu­man­ity is likely to find by Ma­chine Learning

Quick takes on “AI is easy to con­trol”

• So8resNov 28, 2023, 3:41 PM

  23 points

  14

  in reply to: Scott Alexander’s comment on: Apoca­lypse in­surance, and the hardline liber­tar­ian take on AI risk

  Agreed that the proposal is underspecified; my point here is not “look at this great proposal” but rather “from a theoretical angle, risking others’ stuff without the ability to pay to cover those risks is an indirect form of probabilistic theft (that market-supporting coordination mechanisms must address)” plus “in cases where the people all die when the risk is realized, the ‘premiums’ need to be paid out to individuals in advance (rather than paid out to actuaries who pay out a large sum in the event of risk realization)”. Which together yield the downstream inference that society is doing something very wrong if they just let AI rip at current levels of knowledge, even from a very laissez-faire perspective.

  (The “caveats” section was attempting—and apparently failing—to make it clear that I wasn’t putting forward any particular policy proposal I thought was good, above and beyond making the above points.)

Apoca­lypse in­surance, and the hardline liber­tar­ian take on AI risk

Abil­ity to solve long-hori­zon tasks cor­re­lates with want­ing things in the be­hav­iorist sense

How much to up­date on re­cent AI gov­er­nance moves?

Thoughts on the AI Safety Sum­mit com­pany policy re­quests and responses

AI as a sci­ence, and three ob­sta­cles to al­ign­ment strategies

• So8resOct 24, 2023, 8:44 PM

  62 points

  48

  on: So8res’s Shortform

  In relation to my current stance on AI, I was talking with someone who said they’re worried about people putting the wrong incentives on labs. At various points in that convo I said stuff like (quotes are not exact; third paragraph is a present summary rather than a re-articulation of a past utterance):

  “Sure, every lab currently seems recklessly negligent to me, but saying stuff like “we won’t build the bioweapon factory until we think we can prevent it from being stolen by non-state actors” is directionally better than not having any commitments about any point at which they might pause development for any reason, which is in turn directionally better than saying stuff like “we are actively fighting to make sure that the omnicidal technology is open-sourced”.”

  And: “I acknowledge that you see a glimmer of hope down this path where labs make any commitment at all about avoiding doing even some minimal amount of scaling until even some basic test is passed, e.g. because that small step might lead to more steps, and/​or that sort of step might positively shape future regulation. And on my notion of ethics it’s important to avoid stomping on other people’s glimmers of hope whenever that’s feasible (and subject to some caveats about this being tricky to navigate when your hopes are opposed), and I’d prefer people not stomp on that hope.”

  I think that the labs should Just Fucking Stop but I think we should also be careful not to create more pain for the companies that are doing relatively better, even if that better-ness is miniscule and woefully inadequate.

  My conversation partner was like “I wish you’d say that stuff out loud”, and so, here we are.