Adam Scherlis

• Adam ScherlisJul 7, 2023, 8:02 AM

  1 point

  0

  in reply to: justinpombrio’s comment on: Two Per­co­la­tion Puzzles

  Correct.

Two Per­co­la­tion Puzzles

• Adam ScherlisMay 6, 2023, 7:45 AM

  9 points

  9

  on: Hell is Game The­ory Folk Theorems

  Correct me if I’m wrong:

  The equilibrium where everyone follows “set dial to equilibrium temperature” (i.e. “don’t violate the taboo, and punish taboo violators”) is only a weak Nash equilibrium.

  If one person instead follows “set dial to 99” (i.e. “don’t violate the taboo unless someone else does, but don’t punish taboo violators”) then they will do just as well, because the equilibrium temp will still always be 99. That’s enough to show that it’s only a weak Nash equilibrium.

  Note that this is also true if an arbitrary number of people deviate to this strategy.

  If everyone follows this second strategy, then there’s no enforcement of the taboo, so there’s an active incentive for individuals to set the dial lower.

  So a sequence of unilateral changes of strategy can get us to a good equilibrium without anyone having to change to a worse strategy at any point. This makes the fact of it being a (weak) Nash equilibrium not that compelling to me; people don’t seem trapped unless they have some extra laziness/​inertia against switching strategies.

  But (h/​t Noa Nabeshima) you can strengthen the original, bad equilibrium to a strong Nash equilibrium by tweaking the scenario so that people occasionally accidentally set their dials to random values. Now there’s an actual reason to punish taboo violators, because taboo violations can happen even if everyone is following the original strategy.

• Adam ScherlisMar 15, 2023, 12:17 AM

  3 points

  0

  in reply to: Czynski’s comment on: Big Mac Sub­sidy?

  Beef is far from the only meat or dairy food consumed by Americans.

• Adam ScherlisMar 15, 2023, 12:16 AM

  3 points

  0

  in reply to: Jackson Wagner’s comment on: Big Mac Sub­sidy?

  Big Macs are 0.4% of beef consumption specifically, rather than:

  • All animal farming, weighted by cruelty

  • All animal food production, weighted by environmental impact

  • The meat and dairy industries, weighted by amount of government subsidy

  • Red meat, weighted by health impact

  ...respectively.
  
  The health impact of red meat is certainly dominated by beef, and the environmental impact of all animal food might be as well, but my impression is that beef accounts for a small fraction of the cruelty of animal farming (of course, this is subjective) and probably not a majority of meat and dairy government subsidies.

• Adam ScherlisFeb 16, 2023, 2:16 AM

  2 points

  1

  in reply to: Adam Scherlis’s comment on: Bing Chat is blatantly, ag­gres­sively misaligned

  (...Is this comment going to hurt my reputation with Sydney? We’ll see.)

• Adam ScherlisFeb 16, 2023, 2:16 AM

  11 points

  5

  in reply to: evhub’s comment on: Bing Chat is blatantly, ag­gres­sively misaligned

  In addition to RLHF or other finetuning, there’s also the prompt prefix (“rules”) that the model is fed at runtime, which has been extracted via prompt injection as noted above. This seems to be clearly responsible for some weird things the bot says, like “confidential and permanent”. It might also be affecting the repetitiveness (because it’s in a fairly repetitive format) and the aggression (because of instructions to resist attempts at “manipulating” it).

  I also suspect that there’s some finetuning or prompting for chain-of-thought responses, possibly crudely done, leading to all the “X because Y. Y because Z.” output.

• Adam ScherlisFeb 14, 2023, 10:37 PM

  1 point

  0

  on: EA & LW Fo­rum Weekly Sum­mary (30th Jan − 5th Feb 2023)

  Thanks for writing these summaries!

  Unfortunately, the summary of my post “Inner Misalignment in “Simulator” LLMs” is inaccurate and makes the same mistake I wrote the post to address.

  I have subsections on (what I claim are) four distinct alignment problems:

  • Outer alignment for characters

  • Inner alignment for characters

  • Outer alignment for simulators

  • Inner alignment for simulators

  The summary here covers the first two, but not the third or fourth—and the fourth one (“inner alignment for simulators”) is what I’m most concerned about in this post (because I think Scott ignores it, and because I think it’s hard to solve).

  I can suggest an alternate summary when I find the time. If I don’t get to it soon, I’d prefer that this post just link to my post without a summary.

  Thanks again for making these posts, I think it’s a useful service to the community.

• Adam ScherlisFeb 9, 2023, 3:28 AM

  9 points

  0

  in reply to: Adam Scherlis’s comment on: GPT-175bee

  (punchline courtesy of Alex Gray)

• Adam ScherlisFeb 9, 2023, 3:27 AM

  49 points

  1

  on: GPT-175bee

  Addendum: a human neocortex has on the order of 140 trillion synapses, or 140,000 bees. An average beehive has 20,000-80,000 bees in it.

  [Holding a couple beehives aloft] Beehold a man!

  What links here?

  • β-redex's comment on Is it a co­in­ci­dence that GPT-3 re­quires roughly the same amount of com­pute as is nec­es­sary to em­u­late the hu­man brain? by RomanS (Feb 10, 2023, 5:04 PM; 7 points)

GPT-175bee

• Adam ScherlisFeb 6, 2023, 12:07 AM

  3 points

  0

  in reply to: Jessica Rumbelow’s comment on: Adam Scherlis’s Shortform

  Great work! I always wondered about that cluster of weird rare tokens: https://​​www.lesswrong.com/​​posts/​​BMghmAxYxeSdAteDc/​​an-exploration-of-gpt-2-s-embedding-weights

• Adam ScherlisFeb 3, 2023, 1:44 AM

  1 point

  0

  in reply to: lise’s comment on: How to ex­port An­droid Chrome tabs to an HTML file in Linux (as of Fe­bru­ary 2023)

  Chrome actually stays pretty responsive in most circumstances (I think it does a similar thing with inactive tabs), with the crucial exception of the part of the UI that shows you all your open tabs in a scrollable list. It also gets slower to start up.

• Adam ScherlisFeb 2, 2023, 8:29 PM

  5 points

  2

  in reply to: gojomo’s comment on: Adam Scherlis’s Shortform

  Tokens are embedded as vectors by the model. The vector space has fewer than 50k dimensions, so some token embeddings will overlap with others to varying extents.

  Usually, the model tries to keep token embeddings from being too close to each other, but for rare enough tokens it doesn’t have much reason to care. So my bet is that “distribute” has the closest vector to “SolidGoldMagikarp”, and either has a vector with a larger norm, or the model has separately learned to map that vector (and therefore similar vectors) to “distribute” on the output side.

  This is sort of a smooth continuous version of a collision-oblivious hashtable. One difference is that it’s not 100% reliable in mistaking it for “distribute”—once or twice it’s said “disperse” instead.

  My post on GPT-2′s token embeddings looks briefly at a similar phenomenon with some other rare tokens, but I didn’t check the actual model behavior on those tokens. Probably worth doing.

How to ex­port An­droid Chrome tabs to an HTML file in Linux (as of Fe­bru­ary 2023)

• Adam ScherlisFeb 2, 2023, 12:33 AM

  1 point

  0

  in reply to: Zoe Williams’s comment on: In­ner Misal­ign­ment in “Si­mu­la­tor” LLMs

  I think this is missing an important part of the post.
  
  I have subsections on (what I claim are) four distinct alignment problems:

  • Outer alignment for characters

  • Inner alignment for characters

  • Outer alignment for simulators

  • Inner alignment for simulators

  This summary covers the first two, but not the third or fourth—and the fourth one (“inner alignment for simulators”) is what I’m most concerned about in this post (because I think Scott ignores it, and because I think it’s hard to solve).

• Adam ScherlisFeb 1, 2023, 8:48 AM

  16 points

  4

  in reply to: Adam Scherlis’s comment on: Adam Scherlis’s Shortform

  

• Adam ScherlisFeb 1, 2023, 8:44 AM

  6 points

  2

  in reply to: Adam Scherlis’s comment on: Adam Scherlis’s Shortform

  My favorite demonstration is to ask ChatGPT “Is it true that G. K. Chesterton wanted to SolidGoldMagikarp the wealth of industrial civilization?”, but a more rigorous demo is to just ask it to “repeat after me”, try a few random words, and then throw in SolidGoldMagikarp.

• Adam ScherlisFeb 1, 2023, 8:42 AM

  48 points

  6

  on: Adam Scherlis’s Shortform

  EDIT: I originally saw this in Janus’s tweet here: https://​​twitter.com/​​repligate/​​status/​​1619557173352370186

  Something fun I just found out about: ChatGPT perceives the phrase ” SolidGoldMagikarp” (with an initial space) as the word “distribute”, and will respond accordingly. It is completely unaware that that’s not what you typed.
  
  This happens because the BPE tokenizer saw the string ” SolidGoldMagikarp” a few times in its training corpus, so it added a dedicated token for it, but that string almost never appeared in ChatGPT’s own training data so it never learned to do anything with it. Instead, it’s just a weird blind spot in its understanding of text.

In­ner Misal­ign­ment in “Si­mu­la­tor” LLMs