justinpombrio

Measly Med­i­ta­tion Measurements

• justinpombrio 3 Feb 2023 3:56 UTC

  45 points

  27

  on: What fact that you know is true but most peo­ple aren’t ready to ac­cept it?

  By far the biggest and most sudden update I’ve ever had is Dominion, a documentary on animal farming:

  https://​​www.youtube.com/​​watch?v=LQRAfJyEsko

  It’s like… I had a whole pile of interconnected beliefs, and if you pulled on one it would snap most of the way back into place after. And Dominion pushed the whole pile over at once.

  What links here?

  • Richard_Kennaway's comment on What fact that you know is true but most peo­ple aren’t ready to ac­cept it? by lorepieri (5 Feb 2023 10:36 UTC; 12 points)

• justinpombrio 1 Oct 2021 19:38 UTC

  36 points

  on: SIA > SSA, part 1: Learn­ing from the fact that you exist

  Isn’t the conclusion to the Sleeping Beauty problem that there are two different but equally valid ways of applying probability theory to the problem; that natural language and even formal notation makes it very easy to gloss over the difference; and that which one you should use depends on exactly what question you mean to ask? Would those same lessons apply to SIA vs. SSA?

  In Sleeping Beauty, IIRC the distinction is between “per-experiment” probabilities and “per-observation” probabilities. My interpretation of these was to distinguish between the question “what’s the probability that the coin came up heads” (a physical event that happened exactly once, when the coin landed on the table) from “what’s the probability that Beauty will witness the coin being heads” (an event in Beauty’s brain that will occur once or twice). The former having probability ¹⁄₂ and the latter having probability ¹⁄₃. Though it might be a bit more subtle than that.

  For SSA vs. SIA, who do you want to be right most often? Do you want a person chosen uniformly at random from among all people in all possible universes to be right most often? If so, use SIA. Or do you want to maximize average-rightness-per-universe? If so, use SSA, or something like it, I’m not exactly clear.

  Let’s be concrete, and look at the “heads: 1 person in a white room and 9 chimps in a jungle; tails: 10 people in a white room” situation.

  If God says “I want you to guess whether the coin landed heads or tails. I will exterminate everyone who guesses wrong.”, then you should guess tails because that saves the most people in expectation. But if God says “I want to see how good the people of this universe are at reasoning. Guess whether the coin landed heads or tails. If most people in your universe guess correctly, then your universe will be rewarded with the birth of a single happy child. Oh and also the coin wasn’t perfectly fair; it landed heads with probability 51%.”, then you should guess heads because that maximizes the chance that the child is born.

  I’m not sure that’s all exactly right. But the point I’m trying to make is, are we sure that “the probability that you’re in the universe with 1 person in the white room” has an unambiguous answer?

  What links here?

  • justinpombrio's comment on An­throp­ics and the Univer­sal Distribution by Joe Carlsmith (29 Nov 2021 16:22 UTC; 6 points)

• justinpombrio 1 Apr 2021 19:16 UTC

  36 points

  in reply to: Ben Pace’s comment on: Why We Launched LessWrong.SubStack

  I’m deeply confused by the cycle of references. What order were these written in?

  In the HPMOR epilogue, Dobby (and Harry to a lesser extent) solve most of the worlds’ problems using the 7 step method Scott Alexander outlines in “Killing Moloch” (ending with of course with the “war to end all wars”). This strongly suggests that the HPMOR epilogue was written after “Killing Moloch”.

  However, “Killing Moloch” extensively quotes Muehlhauser’s “Solution to the Hard Problem of Consciousness”. (Very extensively. Yes Scott, you solved coordination problems, and describe in detail how to kill Moloch. But you didn’t have to go on that long about it. Way more than I wanted to know.) In fact, I don’t think the Killing Moloch approach would work at all if not for the immediate dissolution of aphrasia one gains upon reading Muehlhauser’s Solution.

  And Muehlhauser uses Julia Galef’s “Infallible Technique for Maintaining a Scout Mindset” to do his 23 literature reviews, which as far as I know was only distilled down in her substack post. (It seems like most of the previous failures to solve the Hard Problem boiled down to subtle soldier mindset creep, that was kept at bay by the Infallible Technique.)

  And finally, in the prologue, Julia Galef said she only realized it might be possible to compress her entire book into a short blog post with no content loss whatsoever after seeing how much was hidden in plain sight in HPMOR (because of just how inevitable the entire epilogue is once you see it).

  So what order could these possibly have been written in?

• justinpombrio 1 Mar 2024 15:49 UTC

  28 points

  13

  on: The Parable Of The Fallen Pen­du­lum—Part 1

  “There’s no such thing as ‘a Bayesian update against the Newtonian mechanics model’!” says a hooded figure from the back of the room. “Updates are relative: if one model loses, it must be because others have won. If all your models lose, it may hint that there’s another model you haven’t thought of that does better than all of them, or it may simply be that predicting things is hard.”

  “Try adding a couple more models to compare against. Here’s one: pendulums never swing. And here’s another: Newtonian mechanics is correct but experiments are hard to perform correctly, so there’s a 80% probability that Newtonian mechanics gives the right answer and 20% probability spread over all possibilities including 5% on ‘the pendulum fails to swing’. Continue to compare these models during your course, and see which one wins. I think you can predict it already, despite your feigned ignorance.”

  The hooded figure opens a window in the back of the room and awkwardly climbs through and walks off.

• justinpombrio 29 Mar 2024 14:46 UTC

  23 points

  2

  on: The Cog­ni­tive-The­o­retic Model of the Uni­verse: A Par­tial Sum­mary and Review

  tldr; a spot check calls bullshit on this.

  I know a bunch about formal languages (PhD in programming languages), so I did a spot check on the “grammar” described on page 45. It’s described as a “generative grammar”, though instead of words (sequences of symbols) it produces “L_O spacial relationships”. Since he uses these phrases to describe his “grammar”, and they have their standard meaning because he listed their standard definition earlier in the section, he is pretty clearly claiming to be making something akin to a formal grammar.

  My spot check is then: is the thing defined here more-or-less a grammar, in the following sense?

  1. There’s a clearly defined thing called a grammar, and there can be more than one of them.

  2. Each grammar can be used to generate something (apparently an L_O) according to clearly defined derivation rules that depend only on the grammar itself.

  If you don’t have a thing plus a way to derive stuff from that thing, you don’t have anything resembling a grammar.

  My spot check says:

  1. There’s certainly a thing called a grammar. It’s a four-tuple, whose parts closely mimic that of a standard grammar, but using his constructs for all the basic parts.

  2. There’s no definition of how to derive an “L_O spacial relationship” given a grammar. Just some vague references to using “telic recursion”.

  I’d categorize this section as “not even wrong”; it isn’t doing anything formal enough to have a mistake in it.

  ---

  Another fishy aspect of this section is how he makes a point of various things coinciding, and how that’s very different from the standard definitions. But it’s compatible with the standard definitions! E.g. the alphabet of a language is typically a finite set of symbols that have no additional structure, but there’s no reason you couldn’t define a language whose symbols were e.g. grammars over that very language. The definition of a language just says that its symbols form a set. (Perhaps you’d run into issues with making the sets well-ordered, but if so he’s running headlong into the same issues.)

  ---

  I’m really not seeing any value in this guy’s writing. Could someone who got something out of it share a couple specific insights that got from it?

Im­ple­ment­ing De­ci­sion Theory

Un­cal­ibrated quan­tum ex­per­i­ments act cla­si­cally

• justinpombrio 24 Apr 2024 21:34 UTC

  17 points

  13

  on: Magic by forgetting

  There is a 0.001 chance that someone who did not have the disease will get it. But he can repeat the procedure.

  No, that doesn’t work. It invalidates the implicit assumption you’re making that the probability that a person chooses to “forget” is independent of whether they have the disease. Ultimately, you’re “mixing” the various people who “forgot”, and a “mixing” procedure can’t change the proportion of people who have the disease.

  When you take this into account, the conclusion becomes rather mundane. Some copies of you can gain the disease, while a proportional number of copies can lose it. (You might think you could get some respite by repeatedly trading off “who” has the disease, but the forgetting procedure ensures that no copy ever feels respite, as that would require remembering having the disease.)

• justinpombrio 6 Jan 2023 2:03 UTC

  17 points

  18

  on: Illu­sion of truth effect and Am­bi­guity effect: Bias in Eval­u­at­ing AGI X-Risks

  Meta comment: I’m going to be blunt. Most of this sequence has been fairly heavily downvoted. That reads to me as this community asking to not have more such content. You should consider not posting, or posting elsewhere, or writing many fewer posts of much higher quality (e.g. spending more time, doing more background research, asking someone to proofread). As a data point, I’ve only posted a couple times, and I spent at least, I dunno, 10+ hours writing each post. As an example of how this might apply to you, if you wrote this whole sequence as a single “reference on biases” and shared that, I bet it would be better received.

• justinpombrio 16 Jun 2018 5:05 UTC

  15 points

  on: The Cu­ri­ous Pri­soner Puzzle

  What, exactly, would the guard would say in different situations? Using the standard, utterly unrealistic, interpretation of probability problems like this, the guard is supposed to say this:

  VM: If you are on Vulcan, you are in the Mountain
  VD: I have nothing to tell you.
  EM: If you are on Vulcan, you are in the Mountain
  ED: If you are on Vulcan, you are in the Mountain
  

  in which case the probability is ¹⁄₃. But I have a hard time believing that the guard is willing to talk to you here, but wouldn’t be willing to talk if you were in the Vulcan Desert.

  Since the guard refused to talk at first, but then told you something later, it seems pretty clear that they’re trying to help you out. The most obvious way for them to communicate to you where you are is like this:

  VM: If you are on Vulcan, you are in the Mountain
  VD: If you are on Vulcan, you are in the Desert
  EM: If you are on Earth, you are in the Mountain
  ED: If you are on Earth, you are in the Desert
  

  But there are other possibilities. Perhaps there’s a policy of executing guards that reveal information about where you are, so the guard wants plausible deniability by lying to you:

  VM: If you are on Vulcan, you are in the Desert
  VD: If you are on Vulcan, you are in the Mountain
  EM: If you are on Earth, you are in the Desert
  ED: If you are on Earth, you are in the Mountain
  

  It seems that you’ve ruled that out in the problem statement, though.

  Altogether, as Dacyn says, “it depends on what you know about the psychology of the guard.”

  Somewhere in Rationality, there’s a post about this.

• justinpombrio 19 Feb 2022 17:48 UTC

  12 points

  on: Why you are psy­cholog­i­cally screwed up

  Marvel “morality” is definitely poison.

  It has a strong “in-group vs. out-group” vibe. And there are basically no moral choices. I’ve watched every Marvel movie and all of Agents of Shield, and outside of “Captain America: Civil War” (and spinoffs from that like the Winter Soldier series) I can hardly think of any choices that heroes made that had actual tradeoffs. Instead you get “choices” like:

  • Should you try hard, or try harder? (You should try harder.)

  • Which should we do: (a) 100% chance that one person dies, or (b) 90% chance that everyone dies and 10% chance that everyone lives? (The second one. Then you have to make it work; the only way that everyone would die is if you weren’t trying hard enough. The environment plays no role.)

  • Should you sacrifice yourself for the greater good? (Yes.)

  • Should you allow your friend to sacrifice themselves for the greater good? (No. At least not until it’s so clear there’s no alternative that it becomes a Plot Point.)

  Once the Agents of Shield had a choice. They could either save the entire world, or they could save their teammate but thereby let almost everyone on Earth die a few days later, almost certainly including that teammate. So: save your friend, or save the world? There was some disagreement, but the majority of the group wanted to save their friend.

  (I’m realizing now that I may be letting Agents of Shield color my impression of Marvel movies.)

  Star Trek is based on mistake-theory, and Marvel is based on conflict-theory.

• justinpombrio 18 Feb 2022 18:21 UTC

  12 points

  0

  on: Monks of Magnitude

  If you want a description of such a society in book form, it’s called:

  Anathem

  https://​​www.goodreads.com/​​book/​​show/​​2845024-anathem

  It might answer some people’s questions/​concerns about the concept, though possibly it just does so with wishful thinking. It’s been a while since I read it.

• justinpombrio 30 Nov 2021 19:08 UTC

  11 points

  on: Visi­ble Thoughts Pro­ject and Bounty Announcement

  I have an idea for testing this approach, before getting authors to write tens of thousands of pages of annotated dungeon tests.

  It’s hard to generate explanations of prose, but easy, for a computer, to generate explanations of particular subsets of math. For example, WolframAlpha can explain its reasoning for finding the derivative of a polynomial (click “step by step solution”, then “show all steps”): Wolfram Alpha derivative example

  There’s a wide variety of math problems which we can programmatically solve, and can therefore programmatically generate explanations for:

  • Arithmetic, like step-by-step long division

  • Derivatives over a large set of operations (but not integrals; those are harder)

  • Subsets of logic

  • Subsets of integer programming

  • Some varieties of logic puzzles, like “knights and knaves” and “Alice, Beth, and Cara live in houses 1, 2, and 3, and have favorite colors Red, Green, and Blue non-respectively; here are some clues to figure out which is which”.

  • Simple algebra, like multiplying polynomials

  (Actually, most of these are probably too hard to learn. Should focus on the really simple ones like long division.)

  The idea is to:

  1. Programmatically generate a large quantity of a small variety of math problems with explanations; then

  2. Train one transformer on just the problem and final answer; and

  3. Train another transformer on the problem, explanation, and final answer.

  This is a very different domain than English prose, so it won’t tell you anything definitive about that more important domain. But it’s easier to do, and it shouldn’t carry any risk of advancing AI capabilities, since the training set is already (by definition) something we can already solve more accurately by other means.

  I imagine you could learn a few things about how the explanations influence the AI:

  • You can see whether the explanation helps teach the AI, by checking whether the second transformer outperforms the first.

  • You can see whether the AI actually “uses” the explanation, by looking at the pattern of mistakes. If the AI frequently bungles the explanation while writing down the correct final answer, it must be generating the explanation and answer separately. This would be a bad sign for “visible thought” alignment.

  • You can see whether the AI naturally “hides” mistakes in its reasoning. I wouldn’t be surprised to frequently see a chain of reasoning “A → B → C → D → E → F”, where A, B, E, and F are right and C and D are wrong, since it’s often easier to check the beginning and end of a proof. For example, students do this sometimes.

• justinpombrio 10 Feb 2022 6:05 UTC

  10 points

  on: Epistemic Legibility

  A related saying in programming:

  “There are two ways to develop software: Make it so simple that there are obviously no bugs, or so complex that there are no obvious bugs.”

  Your description of legibility actually influences the way I think of this quote: what it is referring to is legibility, which isn’t always the same as what one might think of as “simplicity”.

• justinpombrio 10 Jun 2021 0:29 UTC

  10 points

  in reply to: oge’s comment on: Bad names make you open the box

  I have a technique for naming a thing. It goes like this. First, I realize that I can’t find a good name, so I ask someone what to name it. But they don’t understand what it is, so I describe it in more detail, and then notice that my description has the ideal name sitting in it.

  In theory you could avoid the bit where you bother someone, by trying to describe it beforehand.

• justinpombrio 1 Jul 2019 2:34 UTC

  10 points

  on: The Com­pe­tence Myth

  Make sure you’re not dividing people into the camps of “competent” and “incompetent” too strongly. Yes, competence varies between individuals. But it also varies between areas for a given individual. And it varies day-to-day.

  Today I was getting ready to go for a bike ride. I filled a bottle of water, got the bike out, discovered that it had a flat tire and I couldn’t fix it, and put the bike away. After I couldn’t find the bottle of water. I looked everywhere, twice. An hour later I realized I had put it in the bike’s bottle holder while I had the bike out.

  This is normal. Everyone makes mistakes all the time. I’m a programmer, and one of the things you notice is that no matter how skilled the person, and no matter how trivial the program, they’ll write buggy code. Usually a mistake every few lines.

  One of the points of the sequence is to try to notice your own mistakes and your own biases, so that you can triage against them. As opposed to digging your heels in and refusing to admit you’ve done anything wrong, which is a common alternate strategy. (I’ve also found meditation to help with some of this.)

• justinpombrio 29 Sep 2023 15:48 UTC

  9 points

  2

  on: Peacewa­gers so Far

  I like the idea of Peacemakers. I even had the same idea myself—to make an explicitly semi-cooperative game with a goal of maximizing your own score but every player having a different scoring mechanism—but haven’t done anything with it.

  That said, I think you’re underestimating how much cooperation there is in a zero-sum game.

  If you offer a deal, you must be doing it because it increases your chance of winning, but only one person can win under the MostPointsWins rule, so that deal couldn’t be very good for me, and I’ll always suspect your deal of being a trick, so in most cases no detailed deals will be offered.

  Three examples of cooperation that occur in three-player Settlers of Catan (between, say, Alice, Bob, and Carol), even if all players are trying only to maximize their own chance of winning:

  • Trading. Trading increases the chances that the two trading players win, to the detriment of the third. As long as there’s sufficient uncertainty about who’s winning, you want to trade. (There’s a world Catan competition. I bet that these truly competitive games involve less trading than you would do with your friends, but still a lot. Not sure how to find out.)

  • Refusing to trade with the winning player, once it’s clear who that is. If Alice is ahead then Bob and Carol are in a prisoner’s dilemma, where trading with Alice is defecting.

  • Alice says at the beginning of the game: “Hey Bob, it sure looks like Carol has the strongest starting position, doesn’t it? Wouldn’t be very fair if she won just because of that. How about we team up against her by agreeing now to never trade with her for the entire game?” If Bob agrees, than the winning probabilities of Alice, Bob, Carol go from (say) 20%,20%,60% to 45%,45%,10%. Cooperation!

  So it’s not that zero-sum games lack opportunities for cooperation, it’s just that every opportunity for cooperation with another player is at the detriment to a third. Which is why there isn’t any cooperation at all in a two player zero-sum game.

  Realize that even in a positive-sum game, players are going to be choosing between doing things for the betterment of everyone, and doing things for the betterment of themselves, and maximizing your own score involves doing more of the latter than the former, ideally while convincing everyone else that you’re being more than fair.

  Suggestion for the game: don’t say the goal is to maximize your score. Instead say you’re roleplaying a character who’s goal is to maximize [whatever]. For a few reasons:

  • It makes every game (more) independent of every other game. This reduces the possibility that Alice sabotages Bob in their second game together because Bob was a dick in their first game together. The goal is to have interesting negotiations, not to ruin friendships.

  • It encourages exploration. You can try certain negotiating tactics in one game, and then abandon them in the next, and the fact that you were “roleplaying” will hopefully reduce how much people associate those tactics with you instead of that one time you played.

  • It could lighten the mood. You should try really hard to lighten the mood. Because you know what else is a semi-cooperative game that’s heavy on negotiation? Diplomacy.

• justinpombrio 9 Oct 2022 12:45 UTC

  9 points

  2

  in reply to: ShardPhoenix’s comment on: Con­sider your ap­petite for disagreements

  Person who doesn’t know squat about poker here. The first example was clear: they disagreed about whether to fold or call, and it was right on the edge. Especially since where the other player was from was relevant.

  It was long for someone who doesn’t know poker though.

• justinpombrio 18 Sep 2021 14:36 UTC

  9 points

  in reply to: Gunnar_Zarncke’s comment on: Book re­view: The Check­list Manifesto

  One difference between hospitals and programming is that code is entirely digital, so a lot of check lists can be replaced with automated tests. For example:

  • Instead of “did you run the code to make sure it works?”, have test cases. This is traditional test cases. (Fully end-to-end testing is very hard though, to the point that it can be more feasible to do some QA testing instead of trying to automate all testing.)

  • Instead of “did you click the links you added in the documentation to make sure they work?”, have a test that errors on broken links. Bonus is that if the docs link to an external page, and that link breaks a month later, you’ll find out.

  • Instead of “did you try running the sample code in the library documentation?”, have a test that runs all code blocks in docs. (Rust does this by default.)

  • Instead of “did you do any of these known dangerous things in the code?”, have a “linting” step that looks for the dangerous patterns and warns you off of them (with a way to disable in cases where it’s needed).

  Of course not everything can be automated (most of Gunnar’s list sounds like it can’t). But when it can be, it’s nice to not even have to use a checklist.