Vivek Hebbar

• Vivek HebbarJan 8, 2024, 11:19 PM

  LW: 1 AF: 1

  0

  AF

  in reply to: Sam Marks’s comment on: What’s up with LLMs rep­re­sent­ing XORs of ar­bi­trary fea­tures?

  Oops, I misunderstood what you meant by unimodality earlier. Your comment seems broadly correct now (except for the variance thing). I would still guess that unimodality isn’t precisely the right well-behavedness desideratum, but I retract the “directionally wrong”.

• Vivek HebbarJan 8, 2024, 7:02 PM

  LW: 3 AF: 2

  0

  AF

  in reply to: Sam Marks’s comment on: What’s up with LLMs rep­re­sent­ing XORs of ar­bi­trary fea­tures?

  The variance of the multivariate uniform distribution $U\left(\right[0,1]\times [0,1\left]\right)$ is largest along the direction $x_1+x_2$, which is exactly the direction which we would want to represent a AND b.

  The variance is actually the same in all directions. One can sanity-check by integration that the variance is ¹⁄₁₂ both along the axis and along the diagonal.

  In fact, there’s nothing special about the uniform distribution here: The variance should be independent of direction for any N-dimensional joint distribution where the N constituent distributions are independent and have equal variance.^[1]

  The diagram in the post showing that “and” is linearly represented works if the features are represented discretely (so that there are exactly 4 points for 2 binary features, instead of a distribution for each combination). As soon as you start defining features with thresholds like DanielVarga did, the argument stops going through in general, and the claim can become false.

  The stuff about unimodality doesn’t seem relevant to me, and in fact seems directionally wrong.

  1. ^

     I have a not-fully-verbalized proof which I don’t have time to write out

• Vivek HebbarJan 7, 2024, 6:07 AM

  LW: 1 AF: 1

  0

  AF

  on: What’s up with LLMs rep­re­sent­ing XORs of ar­bi­trary fea­tures?

  Maybe models track which features are basic and enforce that these features be more salient

  Couldn’t it just write derivative features more weakly, and therefore not need any tracking mechanism other than the magnitude itself?

• Vivek HebbarOct 11, 2023, 11:58 PM

  8 points

  1

  on: Vivek Heb­bar’s Shortform

  It’s sad that agentfoundations.org links no longer work, leading to broken links in many decision theory posts (e.g. here and here)

• Vivek HebbarOct 11, 2023, 10:21 PM

  3 points

  on: Note-Tak­ing with­out Hid­den Messages

  This will initially boost $\overline{H}$ relative to $\overline{S}$ because it will suddenly be joined to a network with is correctly transmitting $\overline{H}$ but which does not understand $\overline{S}$ at all.

  However, as these networks are trained to equilibrium the advantage will disappear as a steganographic protocol is agreed between the two models. Also, this can only be used once before the networks are in equilibrium.

  Why would it be desirable to do this end-to-end training at all, rather than simply sticking the two networks together and doing no further training? Also, can you clarify what the last sentence means?
  (I have guesses, but I’d rather just know what you meant)

• Vivek HebbarOct 7, 2023, 1:35 AM

  22 points

  2

  in reply to: So8res’s comment on: Re­lated Dis­cus­sion from Thomas Kwa’s MIRI Re­search Experience

  I’ve been asked to clarify a point of fact, so I’ll do so here:

  My recollection is that he probed a little and was like “I’m not too worried about that” and didn’t probe further.

  This does ring a bell, and my brain is weakly telling me it did happen on a walk with Nate, but it’s so fuzzy that I can’t tell if it’s a real memory or not. A confounder here is that I’ve probably also had the conversational route “MIRI burnout is a thing, yikes” → “I’m not too worried, I’m a robust and upbeat person” multiple times with people other than Nate.

  In private correspondence, Nate seems to remember some actual details, and I trust that he is accurately reporting his beliefs. So I’d mostly defer to him on questions of fact here.

  I’m pretty sure I’m the person mentioned in TurnTrout’s footnote. I confirm that, at the time he asked me, I had no recollection of being “warned” by Nate but thought it very plausible that I’d forgotten.

Thomas Kwa’s MIRI re­search experience

• Vivek HebbarSep 27, 2023, 12:38 AM

  6 points

  0

  in reply to: leogao’s comment on: Paper: LLMs trained on “A is B” fail to learn “B is A”

  What’s “denormalization”?

• Vivek HebbarJun 5, 2023, 2:00 AM

  LW: 1 AF: 1

  0

  AF

  in reply to: Eliezer Yudkowsky’s comment on: Mak­ing Nanobots isn’t a one-shot pro­cess, even for an ar­tifi­cial superintelligance

  When you describe the “emailing protein sequences → nanotech” route, are you imagining an AGI with computers on which it can run code (like simulations)? Or do you claim that the AGI could design the protein sequences without writing simulations, by simply thinking about it “in its head”?

In­finite-width MLPs as an “en­sem­ble prior”

[Question] Is EDT cor­rect? Does “EDT” == “log­i­cal EDT” == “log­i­cal CDT”?

• Vivek HebbarApr 28, 2023, 10:05 PM

  8 points

  5

  on: LLMs and com­pu­ta­tion complexity

  Cool! It wrote and executed code to solve the problem, and it got it right.

  Are you using chat-GPT-4? I thought it can’t run code?

• Vivek HebbarApr 28, 2023, 6:37 AM

  6 points

  0

  in reply to: jacob_cannell’s comment on: Con­tra Yud­kowsky on Doom from Foom #2

  Interesting, I find what you are saying here broadly plausible, and it is updating me (at least toward greater uncertainity/​confusion). I notice that I don’t expect the 10x effect, or the Von Neumann effect, to be anywhere close to purely genetic. Maybe some path-dependency in learning? But my intuition (of unknown quality) is that there should be some software tweaks which make the high end of this more reliably achievable.

  Anyway, to check that I understand your position, would this be a fair dialogue?:

  Person: “The jump from chimps to humans is some combination of a 3x scaleup and some algorithmic improvements. Once you have human-level AI, scaling it up 3x and adding a chimp-to-human-jump worth of algorithmic improvement would get you something vastly superhuman, like 30x or 1000x Von Neumann, if not incomparable.”

  Vivek’s model of Jacob: “Nope. The 3x scaleup is the only thing, there wasn’t much algorithmic improvement. The chimp-to-human scaling jump was important because it enabled language/​accumulation, but there is nothing else left like that. There’s nothing practical you can do with 3x human-level compute that would 30x Von Neumann^[1], even if you/​AIs did a bunch of algorithmic research.”

  I find your view more plausible than before, but don’t know what credence to put on it. I’d have more of a take if I properly read your posts.

  1. ^

     I’m not sure how to operationalize this “30x-ing” though. Some candidates:

     - “1000 scientists + 30 Von Neumanns” vs. “1000 scientists + 1 ASI”

     - “1 ASI” vs. “30 Von Neumanns”

     - “100 ASIs” vs. “3000 Von Neumanns”

• Vivek HebbarApr 28, 2023, 5:01 AM

  13 points

  4

  in reply to: jacob_cannell’s comment on: Con­tra Yud­kowsky on Doom from Foom #2

  In your view, who would contribute more to science -- 1000 Einsteins, or 10,000 average scientists?^[1]

  “IQ variation is due to continuous introduction of bad mutations” is an interesting hypothesis, and definitely helps save your theory. But there are many other candidates, like “slow fixation of positive mutations” and “fitness tradeoffs^[2]”.

  Do you have specific evidence for either:

  1. Deleterious mutations being the primary source of IQ variation

  2. Human intelligence “plateauing” around the level of top humans^[3]

  Or do you believe these things just because they are consistent with your learning efficiency model and are otherwise plausible?^[4]

  Maybe you have a very different view of leading scientists than most people I’ve read here? My picture here is not based on any high-quality epistemics (e.g. it includes “second-hand vibes”), but I’ll make up some claims anyway, for you to agree or disagree with:

  • There are some “top scientists” (like Einstein, Dirac, Von Neumann, etc). Within them, much of the variance in fame is incidental, but they are clearly a class apart from merely 96th percentile scientists. 1000 {96%-ile-scientists} would be beaten by 500 {96%-ile-scientists} + 100 Einstein-level scientists.

  • Even within “top scientists” in a field, the best one is more than 3x as intrinsically productive^[5] as the 100th best one.

  1. ^

     I’m like 90% on the Einsteins for theoretical physics, and 60% on the Einsteins for chemistry

  2. ^

     Within this, I could imagine anything from “this gene’s mechanism obviously demands more energy/​nutrients” to “this gene happens to mess up some other random thing, not even in the brain, just because biochemistry is complicated”. I have no idea what the actual prevalence of any of this is.

  3. ^

     What does this even mean? Should the top 1/​million already be within 10x of peak productivity? How close should the smartest human alive be to the peak? Are they nearly free of deleterious mutations?

  4. ^

     I agree that they are consistent with each other and with your view of learning efficiency, but am not convinced of any of them.

  5. ^

     “intrinsic” == assume they have the same resources (like lab equipment and junior scientists if they’re experimentalists)

• Vivek HebbarApr 27, 2023, 11:23 PM

  LW: 4 AF: 2

  2

  AF

  in reply to: Steven Byrnes’s comment on: AI doom from an LLM-plateau-ist perspective

  It would still be interesting to know whether you were surprised by GPT-4′s capabilities (if you have played with it enough to have a good take)

• Vivek HebbarApr 27, 2023, 9:12 PM

  4 points

  2

  in reply to: jacob_cannell’s comment on: $250 prize for check­ing Jake Can­nell’s Brain Efficiency

  Human intelligence in terms of brain arch priors also plateaus

  Why do you think this?

• Vivek HebbarApr 27, 2023, 11:20 AM

  LW: 7 AF: 5

  AF

  in reply to: TurnTrout’s comment on: ricraz’s Shortform

  POV: I’m in an ancestral environment, and I (somehow) only care about the rewarding feeling of eating bread. I only care about the nice feeling which comes from having sex, or watching the birth of my son, or being gaining power in the tribe. I don’t care about the real-world status of my actual son, although I might have strictly instrumental heuristics about e.g. how to keep him safe and well-fed in certain situations, as cognitive shortcuts for getting reward (but not as terminal values).

  Would such a person sacrifice themselves for their children (in situations where doing so would be a fitness advantage)?

• Vivek HebbarApr 27, 2023, 9:32 AM

  42 points

  29

  on: Con­tra Yud­kowsky on Doom from Foom #2

  Isn’t going from an average human to Einstein a huge increase in science-productivity, without any flop increase? Then why can’t there be software-driven foom, by going farther in whatever direction Einstein’s brain is from the average human?

• Vivek Hebbar 27 Apr 2023 8:59 UTC

  3 points

  2

  in reply to: Vivek Hebbar’s comment on: $250 prize for check­ing Jake Can­nell’s Brain Efficiency

  Of course, my argument doesn’t pin down the nature or rate of software-driven takeoff, or whether there is some ceiling. Just that the “efficiency” arguments don’t seem to rule it out, and that there’s no reason to believe that science-per-flop has a ceiling near the level of top humans.

• Vivek Hebbar 27 Apr 2023 8:55 UTC

  13 points

  6

  in reply to: jacob_cannell’s comment on: $250 prize for check­ing Jake Can­nell’s Brain Efficiency

  You could use all of world energy output to have a few billion human speed AGI, or a millions that think 1000x faster, etc.

  Isn’t it insanely transformative to have millions of human-level AIs which think 1000x faster?? The difference between top scientists and average humans seems to be something like “software” (Einstein isn’t using 2x the watts or neurons). So then it should be totally possible for each of the “millions of human-level AIs” to be equivalent to Einstein. Couldn’t a million Einstein-level scientists running at 1000x speed could beat all human scientists combined?
  And, taking this further, it seems that some humans are at least 100x more productive at science than others, despite the same brain constraints. Then why shouldn’t it be possible to go further in that direction, and have someone 100x more productive than Einstein at the same flops? And if this is possible, it seems to me like whatever efficiency constraints the brain is achieving cannot be a barrier to foom, just as the energy efficiency (and supposed learning optimality?) of the average human brain does not rule out Einstein more than 100x-ing them with the same flops.