johnswentworth

• johnswentworthApr 24, 2025, 8:19 PM

  LW: 2 AF: 2

  −2

  AF

  in reply to: johnswentworth’s comment on: Towards a scale-free the­ory of in­tel­li­gent agency

  Note that the same mistake, but with convexity in the other direction, also shows up in the OP:

  Alice and Bob could toss a coin to decide between options #1 and #2, but then they wouldn’t be acting as an EUM (since EUMs can’t prefer a probabilistic mixture of two options to either option individually).

  An EUM can totally prefer a probabilistic mixture of two options to either option individually; this happens whenever utility is convex with respect to resources (e.g. money). For instance, suppose an agent’s utility is u(money) = money^2. I offer this agent a $1 bet on a fair coinflip at even odds, i.e. it gets $0 if it loses and $2 if it wins. The agent takes the bet: the bet offers u = 0.5*0^2 + 0.5*2^2 = 2, while the baseline offers a certainty of $1 which has u = 1.0*1^2 = 1.

• johnswentworthApr 24, 2025, 6:32 PM

  LW: 7 AF: 3

  0

  AF

  in reply to: Richard_Ngo’s comment on: Towards a scale-free the­ory of in­tel­li­gent agency

  This is a tangent, but might be important.

  or a non-EUM with legible goals (like a risk-averse money-maximizer)

  Our prototypical examples of risk-averse money-maximizers are EUMs. In particular, the Kelly bettor is probably the most central example: it maximizes expected log wealth (i.e. log future money). The concavity of the logarithm makes it risk averse: a Kelly bettor will always take a sure payoff over an uncertain outcome with the same expected payoff.

  I bring this up mainly because the wording makes it sound like you’re under the impression that being an EUM is inconsistent with being a risk-averse money-maximizer, in which case you probably have an incorrect understanding of the level of generality of (nontrivial) expected utility maximization, and should probably update toward EUMs being a better model of real agents more often than you previously thought.

• johnswentworthApr 23, 2025, 3:59 PM

  7 points

  0

  in reply to: J Bostock’s comment on: $500 Bounty Prob­lem: Are (Ap­prox­i­mately) Deter­minis­tic Nat­u­ral La­tents All You Need?

  Sounds like you’ve correctly understood the problem and are thinking along roughly the right lines. I expect a deterministic function of $X$ won’t work, though.

  Hand-wavily: the problem is that, if we take the latent to be a deterministic function $\Delta \left(X\right)$, then $P\left[X|\Delta \right(X\left)\right]$ has lots of zeros in it—not approximate-zeros, but true zeros. That will tend to blow up the KL-divergences in the approximation conditions.

  I’d recommend looking for a function $\Delta \left(\Lambda \right)$. Unfortunately that does mean that low entropy of $\Delta \left(\Lambda \right)$given $X$ has to be proven.

• johnswentworthApr 23, 2025, 3:41 PM

  5 points

  0

  in reply to: Alfred Harwood’s comment on: $500 Bounty Prob­lem: Are (Ap­prox­i­mately) Deter­minis­tic Nat­u­ral La­tents All You Need?

  Some details mildly off, but I think you’ve got the big picture basically right.

  Alternatively, if I didn’t tell you the label, you could estimate it from either X_1 or X_2 equally well. This is the other two diagrams.

  Minor clarification here: the other two diagrams say not only that I can estimate the label equally well from either $X_1$ or $X_2$, but that I can estimate the label (approximately) equally well from $X_1$, $X_2$, or the pair $(X_1,X_2)$.

  I think that the full label $\Lambda$ will be an approximate stochastic natural latent.

  I’d have to run the numbers to check that 200 flips is enough to give a high-confidence estimate of $\Lambda$ (in which case 400 flips from the pair of variables will also put high confidence on the same value with high probability), but I think yes.

  But if we consider only the first bit^[1] of the label (which roughly tells us whether the bias is above or below 50% heads) then this bit will be a deterministic natural latent because with reasonably high certainty, you can guess the first bit of $\Lambda$ from $X_1$ or $X_2$.

  Not quite; I added some emphasis. The first bit will (approximately) satisfy the two redundancy conditions, i.e.$X_1\to X_2\to \text{1bit}\left(\Lambda \right)$ and $X_2\to X_1\to \text{1bit}\left(\Lambda \right)$, and indeed will be an approximately deterministic function of $X$. But it won’t (approximately) satisfy the mediation condition $X_1\leftarrow \text{1bit}\left(\Lambda \right)\to X_2$; the two sets of flips will not be (approximately) independent given only the first bit. (At least not to nearly as good an approximation as the original label.)

  That said, the rest of your qualitative reasoning is correct. As we throw out more low-order bits, the mediation condition becomes less well approximated, the redundancy conditions become better approximated, and the entropy of the coarse-grained latent given $X$ falls.

  So to build a proof along these lines, one would need to show that a bit-cutoff can be chosen such that bit_cutoff($\Lambda$) still mediates (to an approximation roughly $ϵ$-ish), while making the entropy of bit_cutoff($\Lambda$) low given $X$.

  I do think this is a good angle of attack on the problem, and it’s one of the main angles I’d try.

  If a latent satisfies the the three natural latents conditions within ${ϵ}_1$, we can always find a (potentially much bigger) ${ϵ}_2$ such that this latent also satisfies the deterministic latent condition, right? This is why you need to specify that the problem is showing that a deterministic natural latent exists with ‘almost the same’ $ϵ$. Does this sound right?

  Yes. Indeed, if we allow large enough $ϵ$ (possibly scaling with system size/​entropy) then there’s always a deterministic natural latent regardless; the whole thing becomes trivial.

• johnswentworthApr 23, 2025, 3:20 PM

  4 points

  0

  in reply to: David Johnston’s comment on: $500 Bounty Prob­lem: Are (Ap­prox­i­mately) Deter­minis­tic Nat­u­ral La­tents All You Need?

  Oh, you’re right. Man, I was really not paying attention before bed last night! Apologies, you deserve somewhat less tired-brain responses than that.

• johnswentworthApr 23, 2025, 5:27 AM

  2 points

  0

  in reply to: David Johnston’s comment on: $500 Bounty Prob­lem: Are (Ap­prox­i­mately) Deter­minis­tic Nat­u­ral La­tents All You Need?

  A natural latent is, by definition, a latent which satisfies two properties. The first is that the observables are IID conditional on the latent, i.e. the common construction you’re talking about. That property by itself doesn’t buy us much of interest, for our purposes, but in combination with the other property required for a natural latent, it buys quite a lot.

$500 Bounty Prob­lem: Are (Ap­prox­i­mately) Deter­minis­tic Nat­u­ral La­tents All You Need?

• johnswentworthApr 21, 2025, 5:33 AM

  2 points

  0

  in reply to: tlevin’s comment on: john­swent­worth’s Shortform

  Probably not going to have a discussion on the topic right now, but out of honest curiosity: did you read the bill?

• johnswentworthApr 20, 2025, 6:20 PM

  2 points

  0

  in reply to: tailcalled’s comment on: Utility Max­i­miza­tion = De­scrip­tion Length Minimization

  Time spent running.

• johnswentworthApr 20, 2025, 5:37 PM

  2 points

  0

  in reply to: tailcalled’s comment on: Utility Max­i­miza­tion = De­scrip­tion Length Minimization

  Note that the “thing which is compressed” is the optimization target, which is not the whole system. So in your example, it probably wouldn’t be the code itself which is compressed, but rather the runtime of the program (holding the output constant), or maybe the runtimes of a bunch of programs on the machine, or something along those lines.

• johnswentworthApr 20, 2025, 3:23 PM

  3 points

  0

  in reply to: XelaP’s comment on: Utility Max­i­miza­tion = De­scrip­tion Length Minimization

  Fixed. Good catch, thanks!

• johnswentworthApr 16, 2025, 3:39 PM

  6 points

  1

  in reply to: uugr’s comment on: john­swent­worth’s Shortform

  This seems like the most load-bearing belief in the full-cynical model; most of your other examples of fakeness rely on it in one way or another [...]

  Nope!

  Even if the base models are improving, it can still be true that most of the progress measured on the benchmarks is fake, and has basically-nothing to do with the real improvements.

  Even if the base models are improving, it can still be true that the dramatic sounding papers and toy models are fake, and have basically-nothing to do with the real improvements.

  Even if the base models are improving, the propaganda about it can still be overblown and mostly fake, and have basically-nothing to do with the real improvements.

  Even if the base models are improving, the people who feel real scared and just want to be validated can still be doing fake work and in fact be mostly useless, and their dynamic can still have basically-nothing to do with the real improvements.

  Just because the base models are in fact improving does not mean that all this other stuff is actually coupled to the real improvement.

• johnswentworthApr 16, 2025, 3:28 PM

  6 points

  −2

  in reply to: Charbel-Raphaël’s comment on: john­swent­worth’s Shortform

  Sure, they are more-than-zero helpful. Heck, in a relative sense, they’d be one of the biggest wins in AI safety to date. But alas, reality does not grade on a curve.

  One has to bear in mind that the words on that snapshot do not all accurately describe reality in the world where SB1047 passes. “Implement shutdown ability” would not in fact be operationalized in a way which would ensure the ability to shutdown an actually-dangerous AI, because nobody knows how to do that. “Implement reasonable safeguards to prevent societal-scale catastrophes” would in fact be operationalized as checking a few boxes on a form and maybe writing some docs, without changing deployment practices at all, because the rules for the board responsible for overseeing these things made it pretty easy for the labs to capture.

  When I discussed the bill with some others at the time, the main takeaway was that the actually-substantive part was just putting any bureaucracy in place at all to track which entities are training models over 10^26 FLOP/​$100M. The bill seemed unlikely to do much of anything beyond that.

  Even if the bill had been much more substantive, it would still run into the standard problems of AI regulation: we simply do not have a way to reliably tell which models are and are not dangerous, so the choice is to either ban a very large class of models altogether, or allow models which will predictably be dangerous sooner or later. The most commonly proposed substantive proxy is to ban models over a certain size, which would likely slow down timelines by a factor of 2-3 at most, but definitely not slow down timelines by a factor of 10 or more.

• johnswentworthApr 15, 2025, 8:23 PM

  2 points

  1

  in reply to: Veedrac’s comment on: john­swent­worth’s Shortform

  The point of listing the problems with their business model is that they need the AGI narrative in order to fuel the investor cash, without which they will go broke at current spend rates. They have cool products, they could probably make a profit if they switched to optimizing for that (which would mean more expensive products and probably a lot of cuts), but not anywhere near the level of profits they’d need to justify the valuation.

• johnswentworthApr 15, 2025, 8:07 PM

  6 points

  4

  in reply to: Veedrac’s comment on: john­swent­worth’s Shortform

  No, the model here is entirely consistent with OpenAI putting out some actual cool products. Those products (under the model) just aren’t on a path to AGI, and OpenAI’s valuation is very much reliant on being on a path to AGI in the not-too-distant future. It’s the narrative about building AGI which is fake.

• johnswentworthApr 15, 2025, 4:09 PM

  6 points

  0

  in reply to: Thane Ruthenis’s comment on: john­swent­worth’s Shortform

  True, but I feel a bit bad about punching that far down.

• johnswentworthApr 15, 2025, 4:00 PM

  9 points

  −1

  in reply to: Charbel-Raphaël’s comment on: john­swent­worth’s Shortform

  SB1047 was a pretty close shot to something really helpful.

  No, it wasn’t. It was a pretty close shot to something which would have gotten a step closer to another thing, which itself would have gotten us a step closer to another thing, which might have been moderately helpful at best.

• johnswentworthApr 15, 2025, 3:45 AM

  4 points

  −1

  in reply to: Joseph Miller’s comment on: john­swent­worth’s Shortform

  Good point, I should have made those two separate bullet points:

  • Then there’s the AI regulation lobbyists. They lobby and stuff, pretending like they’re pushing for regulations on AI, but really they’re mostly networking and trying to improve their social status with DC People. Even if they do manage to pass any regulations on AI, those will also be mostly fake, because (a) these people are generally not getting deep into the bureaucracy which would actually implement any regulations, and (b) the regulatory targets themselves are aimed at things which seem easy to target (e.g. training FLOP limitations) rather than actually stopping advanced AI. The activists and lobbyists are nominally enemies of OpenAI, but in practice they all benefit from pushing the same narrative, and benefit from pretending that everyone involved isn’t faking everything all the time.

  • Also, there’s the AI regulation activists, who e.g. organize protests. Like ~98% of protests in general, such activity is mostly performative and not the sort of thing anyone would end up doing if they were seriously reasoning through how best to spend their time in order to achieve policy goals. Calling it “fake” feels almost redundant. Insofar as these protests have any impact, it’s via creating an excuse for friendly journalists to write stories about the dangers of AI (itself an activity which mostly feeds the narrative, and has dubious real impact).

  (As with the top level, epistemic status: I don’t fully endorse all this, but I think it’s a pretty major mistake to not at least have a model like this sandboxed in one’s head and check it regularly.)

• johnswentworthApr 14, 2025, 6:22 PM

  9 points

  −37

  in reply to: niplav’s comment on: john­swent­worth’s Shortform

  Chris Olah and Dan Murfet in the at-least-partially empirical domain. Myself in the theory domain, though I expect most people (including theorists) would not know what to look for to distinguish fake from non-fake theory work. In the policy domain, I have heard that Microsoft’s lobbying team does quite non-fake work (though not necessarily in a good direction). In the capabilities domain, DeepMind’s projects on everything except LLMs (like e.g. protein folding, or that fast matrix multiplication paper) seem consistently non-fake, even if they’re less immediately valuable than they might seem at first glance. Also Conjecture seems unusually good at sticking to reality across multiple domains.

• johnswentworthApr 14, 2025, 5:50 PM

  148 points

  10

  on: john­swent­worth’s Shortform

  … But It’s Fake Tho

  Epistemic status: I don’t fully endorse all this, but I think it’s a pretty major mistake to not at least have a model like this sandboxed in one’s head and check it regularly.

  Full-cynical model of the AI safety ecosystem right now:

  • There’s OpenAI, which is pretending that it’s going to have full AGI Any Day Now, and relies on that narrative to keep the investor cash flowing in while they burn billions every year, losing money on every customer and developing a product with no moat. They’re mostly a hype machine, gaming metrics and cherry-picking anything they can to pretend their products are getting better. The underlying reality is that their core products have mostly stagnated for over a year. In short: they’re faking being close to AGI.

  • Then there’s the AI regulation activists and lobbyists. They lobby and protest and stuff, pretending like they’re pushing for regulations on AI, but really they’re mostly networking and trying to improve their social status with DC People. Even if they do manage to pass any regulations on AI, those will also be mostly fake, because (a) these people are generally not getting deep into the bureaucracy which would actually implement any regulations, and (b) the regulatory targets themselves are aimed at things which seem easy to target (e.g. training FLOP limitations) rather than actually stopping advanced AI. The activists and lobbyists are nominally enemies of OpenAI, but in practice they all benefit from pushing the same narrative, and benefit from pretending that everyone involved isn’t faking everything all the time.

  • Then there’s a significant contingent of academics who pretend to produce technical research on AI safety, but in fact mostly view their job as producing technical propaganda for the regulation activists and lobbyists. (Central example: Dan Hendrycks, who is the one person I directly name mainly because I expect he thinks of himself as a propagandist and will not be particularly offended by that description.) They also push the narrative, and benefit from it. They’re all busy bullshitting research. Some of them are quite competent propagandists though.

  • There’s another significant contingent of researchers (some at the labs, some independent, some academic) who aren’t really propagandists, but mostly follow the twitter-memetic incentive gradient in choosing their research. This tends to generate paper titles which sound dramatic, but usually provide pretty little conclusive evidence of anything interesting upon reading the details, and very much feed the narrative. This is the main domain of Not Measuring What You Think You Are Measuring and Symbol/​Referent Confusions.

  • Then of course there’s the many theorists who like to build neat toy models which are completely toy and will predictably not generalize useful to real-world AI applications. This is the main domain of Ad-Hoc Mathematical Definitions, the theorists’ analogue of Not Measuring What You Think You Are Measuring.

  • Benchmarks. When it sounds like a benchmark measures something reasonably challenging, it nearly-always turns out that it’s not really measuring the challenging thing, and the actual questions/​tasks are much easier than the pitch would suggest. (Central examples: software eng, GPQA, frontier math.) Also it always turns out that the LLMs’ supposedly-impressive achievement relied much more on memorization of very similar content on the internet than the benchmark designers expected.

  • Then there’s a whole crowd of people who feel real scared about AI (whether for good reasons or because they bought the Narrative pushed by all the people above). They mostly want to feel seen and validated in their panic. They have discussions and meetups and stuff where they fake doing anything useful about the problem, while in fact they mostly just emotionally vibe with each other. This is a nontrivial chunk of LessWrong content, as e.g. Val correctly-but-antihelpfully pointed out. It’s also the primary motivation behind lots of “strategy” work, like e.g. surveying AI researchers about their doom probabilities, or doing timeline forecasts/​models.

  … and of course none of that means that LLMs won’t reach supercritical self-improvement, or that AI won’t kill us, or [...]. Indeed, absent the very real risk of extinction, I’d ignore all this fakery and go about my business elsewhere. I wouldn’t be happy about it, but it wouldn’t bother me any more than all the (many) other basically-fake fields out there.

  Man, I really just wish everything wasn’t fake all the time.

  What links here?

  • The EU Is Ask­ing for Feed­back on Fron­tier AI Reg­u­la­tion (Open to Global Ex­perts)—This Post Breaks Down What’s at Stake for AI Safety by Katalina Hernandez (Apr 22, 2025, 8:39 PM; 57 points)
  • AI #112: Re­lease the Everything by Zvi (Apr 17, 2025, 3:10 PM; 41 points)