Garrett Baker

Karma: 4,642

I have signed no contracts or agreements whose existence I cannot mention.

They thought they found in numbers, more than in fire, earth, or water, many resemblances to things which are and become; thus such and such an attribute of numbers is justice, another is soul and mind, another is opportunity, and so on; and again they saw in numbers the attributes and ratios of the musical scales. Since, then, all other things seemed in their whole nature to be assimilated to numbers, while numbers seemed to be the first things in the whole of nature, they supposed the elements of numbers to be the elements of all things, and the whole heaven to be a musical scale and a number.

Metaph. A. 5, 985 b 27–986 a 2.

Garrett Baker Apr 4, 2025, 8:17 PM
1 point
0
in reply to: Jeremy Gillen’s comment on: Changing my mind about Christiano’s malign prior argument

I’m not sure what the type signature of $L_{A}$ is, or what it means to “not take into account $M$ ’s simulation”

I know you know about logical decision theory, and I know you know its not formalized, and I’m not going to be able to formalize it in a LessWrong comment, so I’m not sure what you want me to say here. Do you reject the idea of logical counterfactuals? Do you not see how they could be used here?

I think you’ve misunderstood me entirely. Usually in a decision problem, we assume the agent has a perfectly true world model, and we assume that it’s in a particular situation (e.g. with omega and knowing how omega will react to different actions). But in reality, an agent has to learn which kind of world its in using an inductor. That’s all I meant by “get its beliefs”.

Because we’re talking about priors and their influence, all of this is happening inside the agent’s brain. The agent is going about daily life, and thinks “hm, maybe there is an evil demon simulating me who will give me −10¹⁰10^10 utility if I don’t do what they want for my next action”. I don’t see why this is obviously ill-defined without further specification of the training setup.

Garrett Baker Apr 4, 2025, 8:06 PM
4 points
0
in reply to: Cole Wyeth’s comment on: AI 2027: What Superintelligence Looks Like
Can you give something specific? It seems like pretty much every statement has a footnote grounding the relevant high-level claim in low-level indicators, and in cases where that’s not the case, those predictions often seem clear derivatives of precise claims in eg their compute forecast

Garrett Baker Apr 4, 2025, 7:18 PM
2 points
−2
in reply to: Cole Wyeth’s comment on: AI 2027: What Superintelligence Looks Like
I mean its not like they shy away from concrete predictions. Eg their first prediction is

We forecast that mid-2025 agents will score 85% on SWEBench-Verified.

Edit: oh wait nevermind their first prediction is actually

Specifically, we forecast that they score 65% on the OSWorld benchmark of basic computer tasks (compared to 38% for Operator and 70% for a typical skilled non-expert human).

Garrett Baker Apr 4, 2025, 7:05 PM
2 points
0
in reply to: leogao’s comment on: leogao’s Shortform
The closing off of China after/during Tinamen square I don’t think happened after a transition of power, though I could be mis-remembering. See also the one-child policy, which I also don’t think happened during a power transition (allowed for 2 children in 2015, then removed all limits in 2021, while Xi came to power in 2012).

I agree the zero-covid policy change ended up being slow. I don’t know why it was slow though, I know a popular narrative is that the regime didn’t want to lose face, but one fact about China is the reason why many decisions are made is highly obscured. It seems entirely possible to me there were groups (possibly consisting of Xi himself) who believed zero-covid was smart. I don’t know much about this though.

I will also say this is one example of china being abnormally slow of many examples of them being abnormally fast, and I think the abnormally fast examples win out overall.

Mao is kind of an exception, but thats because he had so much power that it was impossible to challenge his authority even when he messed up

Ish? The reason he pursued the cultural revolution was because people were starting to question his power, after the great leap forward, but yeah he could be an outlier. I do think that many autocracies are governed by charismatic & powerful leaders though, so not that much an outlier.

Garrett Baker Apr 4, 2025, 6:37 PM
2 points
0
in reply to: Jeremy Gillen’s comment on: Changing my mind about Christiano’s malign prior argument
Let $M$ be an agent which can be instantiated in a much simpler world and has different goals from our limited Bayesian agent $A$ . We say $M$ is malign with respect to $A$ if $p (q | O) < p (q_{M, A} | O)$ where $q$ is the “real” world and $q_{M, A}$ is the world where $M$ has decided to simulate all of $A$ ’s observations for the purpose of trying to invade their prior.

Now what influences $p (q_{M, A} | O)$ ? Well $M$ will only simulate all of $A$ ’s observations if it expects this will give it some influence over $A$ . Let $L_{A}$ be an unformalized logical counterfactual operation that $A$ could make.

Then $p (q_{M, A} | O, L_{A})$ is maximal when $L_{A}$ takes into account $M$ ‘s simulation, and $0$ when $L_{A}$ doesn’t take into account $M$ ‘s simulation. In particular, if $L_{A, \neg M}$ is a logical counterfactual which doesn’t take $M$ ’s simulation into account, then

$p (q_{M, A} | O, L_{A, \neg M}) = 0 < p (q | O, L_{A, \neg M})$ So the way in which the agent “gets its beliefs” about the structure of the decision theory problem is via these logical-counterfactual-conditional operations, same as in causal decision theory, and same as in evidential decision theory.

Garrett Baker Apr 4, 2025, 5:25 PM
2 points
0
in reply to: Jeremy Gillen’s comment on: Changing my mind about Christiano’s malign prior argument

no, I am not going to do what the evil super-simple-simulators want me to do because they will try to invade my prior iff (I would act like they have invaded my prior iff they invade my prior)

Garrett Baker Apr 4, 2025, 5:17 PM
2 points
−3
in reply to: Cole Wyeth’s comment on: AI 2027: What Superintelligence Looks Like
This seems a pretty big backpedal from “I expect this to start not happening right away.”

Garrett Baker Apr 4, 2025, 5:15 PM
2 points
0
in reply to: Jeremy Gillen’s comment on: Changing my mind about Christiano’s malign prior argument
My world model would have a loose model of myself in it, and this will change which worlds I’m more or less likely to be found in. For example, a logical decision theorist, trying to model omega, will have very low probability that omega has predicted it will two box.

Garrett Baker Apr 4, 2025, 4:59 PM
15 points
0
in reply to: leogao’s comment on: leogao’s Shortform
Autarchies, including China, seem more likely to reconfigure their entire economic and social systems overnight than democracies like the US, so this seems false.

Garrett Baker Apr 4, 2025, 4:55 PM
2 points
0
in reply to: Jeremy Gillen’s comment on: Changing my mind about Christiano’s malign prior argument
Oh my point wasn’t against solomonoff in general, maybe more crisply, my clam is different decision theories will find different “pathologies” in the solomonoff prior, and in particular for causal and evidential decision theorists, I could totally buy the misaligned prior bit, and I could totally buy, if formalized, the whole thing rests on the interaction between bad decision theory and solomonoff.

Garrett Baker Apr 4, 2025, 8:09 AM
4 points
2
in reply to: Lucius Bushnaq’s comment on: Changing my mind about Christiano’s malign prior argument
I think I mostly agree with this, I think things possibly get more complicated when you throw decision theory into the mix. I think it unlikely I’m being adversarially simulated in part. I could believe that such malign prior problems are actually decision theory problems much more than epistemic problems. Eg “no, I am not going to do what the evil super-simple-simulators want me to do because they will try to invade my prior iff (I would act like they have invaded my prior iff they invade my prior)”.

Lighthaven Sequences Reading Group #29 (Tuesday 04/08)

Jozdien, Aella, Ronny Fernandez, Ben Pace, Garrett Baker, Jozdien and orthonormal

Apr 4, 2025, 1:16 AM

7 points

0 comments2 min readLW link

Garrett Baker Apr 2, 2025, 8:09 PM
4 points
0
in reply to: Nathan Helm-Burger’s comment on: Towards a scale-free theory of intelligent agency
I’m curious about what neuroscience evidence you’re thinking of which supports that model.

Garrett Baker Apr 1, 2025, 1:04 AM
5 points
2
in reply to: CapResearcher’s comment on: CapResearcher’s Shortform
I’m curious which model it was. Can you post some quotes? Especially after the mask dropped?

Garrett Baker Mar 31, 2025, 3:11 PM
5 points
0
in reply to: M. Y. Zuo’s comment on: adamzerner’s Shortform
That is the wrong question to ask. By their nature the result of experiments is unknown. The bar is whether or not in expectation and on the margin do they provide positive externalities, and the answer is clearly yes.

Garrett Baker Mar 29, 2025, 12:32 AM
6 points
1
in reply to: plex’s comment on: ete’s Shortform
Note the error bars in the original

Garrett Baker Mar 28, 2025, 7:47 PM
2 points
0
in reply to: ank’s comment on: Share AI Safety Ideas: Both Crazy and Not. №2

After that I was writing shorter posts but without long context the things I write are very counterintuitive. So they got ruined)

This sounds like a rationalization. It seems much more likely the ideas just aren’t that high quality if you need a whole hour for a single argument that couldn’t possibly be broken up into smaller pieces that don’t suck.

Edit: Since if the long post is disliked, you can say “well they just didn’t read it”, and if the short post is disliked you can say “well it just sucks because its small”. Meanwhile, it should in fact be pretty surprising you don’t have any interesting or novel or useful insights in your whole 40 minute post which can’t be explained in a reasonable length of blog post time.

Garrett Baker Mar 28, 2025, 7:15 PM
5 points
3
in reply to: ank’s comment on: Share AI Safety Ideas: Both Crazy and Not. №2

usually if you randomly get a downvote early instead of an upvote, so your post has “-1” karma now, then no one else will open or read it

I will say that I often do read −1 downvoted posts, I will also say that much of the time it is deserved, despite how noisy a signal it may be.

Some of my articles take 40 minutes to read, so it can be anything, downvotes give me zero information and just demotivate more and more.

I think you should try writing shorter posts. Both for your sake (so you get more targeted information), and for the readers’ sake.

Garrett Baker Mar 28, 2025, 7:10 PM
2 points
0
in reply to: sarahconstantin’s comment on: sarahconstantin’s Shortform

https://www.science.org/content/blog-post/alkanes-mars there are alkanes—big organic molecules—on Mars. these can be produced by abiotic processes, but usually that makes shorter chains than these. so....life? We Shall See.

Very exciting! I think the biggest “loophole” here is probably that they used a novel technique for detection, maybe if we used that technique more we would have to update the view that such big molecules are so unlikely to be produced non-biologically.

Garrett Baker Mar 28, 2025, 7:01 PM
4 points
0
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
I’m a bit skeptical, there’s a reasonable amount of passed-down wisdom I’ve heard claiming (I think justifiably) that
1. If you write messy code, and say “I’ll clean it later” you probably won’t. So insofar as you eventually want to discover something others build upon, you should write it clean from the start.
2. Clean code leads to easier extensibility, which seems pretty important eg if you want to try a bunch of different small variations on the same experiment.
3. Clean code decreases the number of bugs and the time spent debugging. This seems especially useful insofar as you are trying to rule-out hypotheses with high confidence, or prove hypotheses with high confidence.
4. Generally (this may be double-counting 2 and 3), paradoxically, clean code is faster rather than dirty code.
You say you came from a more SWE based paradigm though, so you probably know all this already.

Garrett Baker

Lighthaven Se­quences Read­ing Group #29 (Tues­day 04/​08)

Lighthaven Sequences Reading Group #29 (Tuesday 04/08)