My blog is here. You can subscribe for new posts there.
My personal site is here.
My X/Twitter is here
You can contact me using this form.
L Rudolf L
Karma: 2,233
I will admit I’m not an expert here. The intuition behind this is that if you grant extreme performance at mathsy things very soon, it doesn’t seem unreasonable that the AIs will make some radical breakthrough in the hard sciences surprisingly soon, while still being bad at many other things. In the scenario, note that it’s a “mathematical framework” (implicitly a sufficiently big advance in what we currently have such that it wins a Nobel) but not the final theory of everything, and it’s explicitly mentioned empirical data bottlenecks it.
Thanks for these speculations on the longer-term future!
while I do think Mars will be exploited eventually, I expect the moon to be first for serious robotics effort
Maybe! My vague Claude-given sense is that the Moon is surprisingly poor in important elements though.
not being the fastest amongst them all (because replicating a little better will usually only get a little advantage, not an utterly dominant one), combined with a lot of values being compatible with replicating fast, so value alignment/intent alignment matters more than you think
This is a good point! However, more intelligence in the world also means we should expect competition to be tighter, reducing the amount of slack by which you can deviate from the optimal. In general, I can see plausible abstract arguments for the long-run equilibrium being either Hansonian zero-slack Malthusian competition or absolute unalterable lock-in.
Given no nationalization of the companies has happened, and they still have large freedoms of action, it’s likely that Google Deepmind, OpenAI and Anthropic have essentially supplanted the US as the legitimate government, given their monopolies on violence via robots.
I expect the US government to be competent enough to avoid being supplanted by the companies. I think politicians, for all their flaws, are pretty good at recognising a serious threat to their power. There’s also only one government but several competing labs.
(Note that the scenario doesn’t mention companies in the mid and late 2030s)
the fact that EA types got hired to some of the most critical positions on AI was probably fairly critical in this timeline for preventing the worst outcomes from the intelligence curse from occurring.
In this timeline, a far more important thing is the sense among American political elite that they are freedom-loving people and that they should act in accordance with that, and a similar sense among Chinese political elite that they are a civilised people and that Chinese civilisational continuity is important. A few EAs in government, while good, will find it difficult to match the impact of the cultural norms that a country’s leaders inherit and that proscribe their actions.
For example: I’ve been reading Christopher Brown’s Moral Capital recently, which looks at how opposition to slavery rose to political prominence in 1700s Britain. It claims that early strong anti-slavery attitudes were more driven by a sense that slavery was insulting to Britons’ sense of themselves as a uniquely liberal people, than by arguments about slave welfare. At least in that example, the major constraint on the treatment of a powerless group of people seems to have been in large part the political elite managing its own self-image.
A History of the Future, 2025-2040
I built this a few months ago: https://github.com/LRudL/devcon
Definitely not production-ready and might require some “minimal configuration and tweaking” to get working.
Includes a “device constitution” that you set; if you visit a website, Claude will judge whether the page follows that written document, and if not it will block you, and the only way past it is winning a debate with it about why your website visit is in-line with your device constitution.
I found it too annoying but some of my friends liked it.
However, I think there is a group of people who over-optimize for Direction and neglect the Magnitude. Increasing Magnitude often comes with the risk of corrupting the Direction. For example, scaling fast often makes it difficult to hire only mission-aligned people, and it requires you to give voting power to investors that prioritizes profit. To increase Magnitude can therefore feel risky, what if I end up working at something that is net-negative for the world? Therefore it might be easier for one’s personal sanity to optimize for Direction, to do something that is unquestionably net-positive. But this is the easy way out, and if you want to have the highest expected value of your Impact, you cannot disregard Magnitude.
You talk here about an impact/direction v ambition/profit tradeoff. I’ve heard many other people talking about this tradeoff too. I think it’s overrated; in particular, if you’re constantly having to think about it, that’s a bad sign.
It’s rare that you have a continuous space of options between lots of impact and low profit, and low/negative impact and high profit.
If you do have such a continuous space of options then I think you are often just screwed and profit incentives will win.
The really important decision you make is probably a discrete choice: do you start an org trying to do X, or an org trying to do Y? Usually you can’t (and even if you can, shouldn’t) try to interpolate between these things, and making this high-level strategy call will probably shape your impact more than any later finetuning of parameters within that strategy.
Often, the profit incentives point towards the more-obvious, gradient-descent-like path, which is usually very crowded and leads to many “mediocre” outcomes (e.g. starting a $10M company), but the biggest things come from doing “Something Else Which Is Not That” (as is said in dath ilan). For example, SpaceX (ridiculously hard and untested business proposition) and Facebook (started out seeing very small and niche and with no clue of where the profit was).
Instead, I think the real value of doing things that are startup-like comes from:
The zero-to-one part of Peter Thiel’s zero-to-one v one-to-n framework: the hardest, progress-bottlenecking things usually look like creating new things, rather than scaling existing things. For example, there is very little you can do today in American politics that is as impactful or reaches as deep into the future as founding America in the first place.
In the case of AI safety: neglectedness. Everyone wants to work at a lab instead, humans are too risk averse in general, etc. (I’ve heard many people in AI safety say that neglectedness is overrated. There are arguments like this one that replaceability/neglectedness considerations aren’t that major: job performance is heavy-tailed, hiring is hard for orgs, etc. But such arguments seem like weirdly myopic parameter-fiddling, at least when the alternative is zero-to-one things like discussed above. Starting big things is in fact big. Paradigm shifts matter because they’re the frame that everything else takes place in. You either see this or you don’t.)
To the extent you think the problem is about economic incentives or differential progress, have you considered getting your hands dirty and trying to change the actual economy or the direction of the tech tree? There are many ways to do this, including some types of policy and research. But I think the AI safety scene has a cultural bias towards things that look like research or information-gathering, and away from being “builders” in the Silicon Valley sense. One of the things that Silicon Valey does get right is that being a builder is very powerful. If the AI debate comes down to a culture/influence struggle between anti-steering, e/acc-influenced builder types and pro-steering EA-influenced academic types, it doesn’t look good for the world.
Thanks for the heads-up, that looks very convenient. I’ve updated the post to link to this instead of the scraper repo on GitHub.
As far as I know, my post started the recent trend you complain about.
Several commenters on this thread (e.g. @Lucius Bushnaq here and @MondSemmel here) mention LessWrong’s growth and the resulting influx of uninformed new users as the likely cause. Any such new users may benefit from reading my recently-curated review of Planecrash, the bulk of which is about summarising Yudkowsky’s worldview.
i continue to feel so confused at what continuity led to some users of this forum asking questions like, “what effect will superintelligence have on the economy?” or otherwise expecting an economic ecosystem of superintelligences
If there’s decision-making about scarce resources, you will have an economy. Even superintelligence does not necessarily imply infinite abundance of everything, starting with the reason that our universe only has so many atoms. Multipolar outcomes seem plausible under continuous takeoff, which the consensus view in AI safety (as I understand it) sees as more likely than fast takeoff. I admit that there are strong reasons for thinking that the aggregate of a bunch of sufficiently smart things is agentic, but this isn’t directly relevant for the concerns about humans within the system in my post.
a value-aligned superintelligence directly creates utopia
In his review of Peter Singer’s commentary on Marx, Scott Alexander writes:
[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.
Peter Thiel might call this “indefinite optimism”: delay all planning or visualisation because there’s some later point where it’s trusted things will all sort themselves out. Now, if you think that takeoff will definitely be extremely hard and the resulting superintelligence will effortlessly take over the world, then obviously it makes sense to focus on what that superintelligence will want to do. But what if takeoff lasts months or years or decades? (Note that there can be lots of change even within months if the stakes look extreme to powerful actors!) Aren’t you curious about what an aligned superintelligence will end up deciding about society and humans? Are you so sure about the transition period being so short and the superintelligence being so unitary and multipolar outcomes being so unlikely that we’ll never have to worry about problems downstream of the incentive issues and competitive pressures that I discuss (which Beren recently had an excellent post on)? Are you so sure that there is not a single interesting, a priori deducible fact about the superintelligent economy beyond “a singleton is in charge and everything is utopia”?
The bottlenecks to compute production are constructing chip fabs; electricity; the availability of rare earth minerals.
Chip fabs and electricity generation are capital!
Right now, both companies have an interest in a growing population with growing wealth and are on the same side. If the population and its buying power begins to shrink, they will be in an existential fight over the remainder, yielding AI-insider/AI-outsider division.
Yep, AI buying power winning over human buying power in setting the direction of the economy is an important dynamic that I’m thinking about.
I also think the AI labor replacement is initially on the side of equality. [...] Now, any single person who is a competent user of Claude can feasibly match the output of any traditional legal team, [...]. The exclusive access to this labor is fundamental to the power imbalance of wealth inequality, so its replacement is an equalizing force.
Yep, this is an important point, and a big positive effect of AI! I write about this here. We shouldn’t lose track of all the positive effects.
Great post! I’m also a big (though biased) fan of Owain’s research agenda, and share your concerns with mech interp.
I’m therefore coining the term “prosaic interpretability”—an approach to understanding model internals [...]
Concretely, I’ve been really impressed by work like Owain Evans’ research on the Reversal Curse, Two-Hop Curse, and Connecting the Dots[3]. These feel like they’re telling us something real, general, and fundamental about how language models think. Despite being primarily empirical, such work is well-formulated conceptually, and yields gearsy mental models of neural nets, independently of existing paradigms.
[emphasis added]
I don’t understand how the papers mentioned are about understanding model internals, and as a result I find the term “prosaic interpretability” confusing.
Some points that are relevant in my thinking (stealing a digram from an unpublished draft of mine):
the only thing we fundamentally care about with LLMs is the input-output behaviour (I-O)
now often, a good way to study the I-O map is to first understand the internals M
but if understanding the internals M is hard but you can make useful generalising statements about the I-O, then you might as well skip dealing with M at all (c.f. psychology, lots of econ, LLM papers like this)
the Owain papers you mention seem to me to make 3 distinct types of moves, in this taxonomy:
finding some useful generalising statement about the I-O map behaviour (potentially conditional on some property of the training data) (e.g. the reversal curse)
creating a modified model M’ from M via fine-tuning on same data (but again, not caring about what the data actually does to the internals)
(far less centrally than the above!) speculating about what the internal structure that causes the behavioural patterns above might be (e.g. that maybe models trained on “A=B” learn to map representation(A) --> representation(B) in some MLP, instead of learning the general rule that A and B are the same thing and representing them internally as such)
So overall, I don’t think the type of work you mention is really focused on internals or interpretability at all, except incidentally in minor ways. (There’s perhaps a similar vibe difference here to category theory v set theory: the focus being relations between (black-boxed) objects, versus the focus being the internals/contents of objects, with relations and operations defined by what they do to those internals)
I think thinking about internals can be useful—see here for a Neel Nanda tweet arguing the reversal curse if obvious if you understand mech interp—but also the blackbox research often has a different conceptual frame, and is often powerful specifically when it can skip all theorising about internals while still bringing true generalising statements about models to the table.
And therefore I’d suggest a different name than “prosaic interpretability”. “LLM behavioural science”? “Science of evals”? “Model psychology”? (Though I don’t particularly like any of these terms)
If takeoff is more continuous than hard, why is it so obvious that there exists exactly one superintelligence rather than multiple? Or are you assuming hard takeoff?
Also, your post writes about “labor-replacing AGI” but writes as if the world it might cause near-term lasts eternally
If things go well, human individuals continue existing (and humans continue making new humans, whether digitally or not). Also, it seems more likely than not that fairly strong property rights continue (if property rights aren’t strong, and humans aren’t augmented to be competitive with the superintelligences, then prospects for human survival seem weak since humans’ main advantage is that they start out owning a lot of the stuff—and yes, that they can shape the values of the AGI, but I tentatively think CEV-type solutions are neither plausible nor necessarily desirable). The simplest scenario is that there is continuity between current and post-singularity property ownership (especially if takeoff is slow and there isn’t a clear “reset” point). The AI stuff might get crazy and the world might change a lot as a result, but these guesses, if correct, seem to pin down a lot of what the human situation looks like.
I already added this to the start of the post:
Edited to add: The main takeaway of this post is meant to be: Labour-replacing AI will shift the relative importance of human v non-human factors of production, which reduces the incentives for society to care about humans while making existing powers more effective and entrenched. Many people are reading this post in a way where either (a) “capital” means just “money” (rather than also including physical capital like factories and data centres), or (b) the main concern is human-human inequality (rather than broader societal concerns about humanity’s collective position, the potential for social change, and human agency).
However:
perhaps you should clarify that you aren’t trying to argue that saving money to spend after AGI is a good strategy, you agree it’s a bad strategy
I think my take is a bit more nuanced:
in my post, I explicitly disagree with focusing purely on getting money now, and especially oppose abandoning more neglected ways of impacting AI development in favour of ways that also optimise for personal capital accumulation (see the start of the takeaways section)
the reason is that I think now is a uniquely “liquid” / high-leverage time to shape the world through hard work, especially because the world might soon get much more locked-in and because current AI makes it easier to do things
(also, I think modern culture is way too risk averse in general, and worry many people will do motivated reasoning and end up thinking they should accept the quant finance / top lab pay package for fancy AI reasons, when their actual reason is that they just want that security and status for prosaic reasons, and the world would benefit most from them actually daring to work on some neglected impactful thing)
however, it’s also true that money is a very fungible resource, and we’re heading into very uncertain times where the value of labour (most people’s current biggest investment) looks likely to plummet
if I had to give advice to people who aren’t working on influencing AI for the better, I’d focus on generically “moving upwind” in terms of fungible resources: connections, money, skills, etc. If I had to pick one to advise a bystander to optimise for, I’d put social connections above money—robust in more scenarios (e.g. in very politicky worlds where money alone doesn’t help), has deep value in any world where humans survive, in post-labour futures even more likely to be a future nexus of status competition, and more life-affirming and happiness-boosting in the meantime
This is despite agreeing with the takes in your earlier comment. My exact views in more detail (comments/summaries in square brackets):
The post-AGI economy might not involve money, it might be more of a command economy. [yep, this is plausible, but as I write here, I’m guessing my odds on this are lower than yours—I think a command economy with a singleton is plausible but not the median outcome]
Even if it involves money, the relationship between how much money someone has before and how much money they have after might not be anywhere close to 1:1. For example: [loss of control, non-monetary power, destructive war] [yep, the capital strategy is not risk-free, but this only really applies re selfish concerns if there are better ways to prepare for post-AGI; c.f. my point about social connections above]
Even if saving money through AGI converts 1:1 into money after the singularity, it will probably be worth less in utility to you
[because even low levels of wealth will max out personal utility post-AGI] [seems likely true, modulo some uncertainty about: (a) utility from positional goods v absolute goods v striving, and (b) whether “everyone gets UBI”-esque stuff is stable/likely, or fails due to despotism / competitive incentives / whatever]
[because for altruistic goals the leverage from influencing AI now is probably greater than leverage of competing against everyone else’s saved capital after AGI] [complicated, but I think this is very correct at least for individuals and most orgs]
Regarding:
you are taking it to mean “we’ll all be living in egalitarian utopia after AGI” or something like that
I think there’s a decent chance we’ll live in a material-quality-of-life-utopia after AGI, assuming “Things Go Fine” (i.e. misalignment / war / going-out-with-a-whimper don’t happen). I think it’s unlikely to be egalitarian in the “everyone has the same opportunities and resources”, for the reasons I lay out above. There are lots of valid arguments for why, if Things Go Fine, it will still be much better than today despite that inequality, and the inequality might not practically matter very much because consumption gets maxed out etc. To be clear, I am very against cancelling the transhumanist utopia because some people will be able to buy 30 planets rather than just a continent. But there are some structural things that make me worried about stagnation, culture, and human relevance in such worlds.
In particular, I’d be curious to hear your takes about the section on state incentives after labour-replacing AI, which I don’t think you’ve addressed and which I think is fairly cruxy to why I’m less optimistic than you about things going well for most humans even given massive abundance and tech.
For example:
Currently big companies struggle to hire and correctly promote talent for the reasons discussed in my post, whereas AI talent will be easier to find/hire/replicate given only capital & legible info
To the extent that AI ability scales with resources (potentially boosted by inference-time compute, and if SOTA models are no longer available to the public), then better-resourced actors have better galaxy brains
Superhuman intelligence and organisational ability in AIs will mean less bureaucratic rot and communication bandwidth problems in large orgs, compared to orgs made out of human brain -sized chunks, reducing the costs of scale
Imagine for example the world where software engineering is incredibly cheap. You can start a software company very easily, yes, but Google can monitor the web for any company that makes revenue off of software, instantly clone the functionality (because software engineering is just a turn-the-crank-on-the-LLM thing now) and combine it with their platform advantage and existing products and distribution channels. Whereas right now, it would cost Google a lot of precious human time and focus to try to even monitor all the developing startups, let alone launch a competing product for each one. Of course, it might be that Google itself is too bureaucratic and slow to ever do this, but someone else will then take this strategy.
C.f. the oft-quoted thing about how the startup challenge is getting to distribution before the incumbents get to distribution. But if the innovation is engineering, and the engineering is trivial, how do you get time to get distribution right?
(Interestingly, as I’m describing it above the most key thing is not so much capital intensivity, and more just that innovation/engineering is no longer a source of differential advantage because everyone can do it with their AIs really well)
There’s definitely a chance that there’s some “crack” in this, either from the economics or the nature of AI performance or some interaction. In particular, as I mentioned at the end, I don’t think modelling the AI as an approaching blank wall of complete perfect intelligence all-obsoleting intelligence is the right model for short to medium -term dynamics. Would be very curious if you have thoughts.
Note, firstly, that money will continue being a thing, at least unless we have one single AI system doing all economic planning. Prices are largely about communicating information. If there are many actors and they trade with each other, the strong assumption should be that there are prices (even if humans do not see them or interact with them). Remember too that however sharp the singularity, abundance will still be finite, and must therefore be allocated.
Though yes, I agree that a superintelligent singleton controlling a command economy means this breaks down.
However it seems far from clear we will end up exactly there. The finiteness of the future lightcone and the resulting necessity of allocating “scarce” resources, the usefulness of a single medium of exchange (which you can see as motivated by coherence theorems if you want), and trade between different entities all seem like very general concepts. So even in futures that are otherwise very alien, but just not in the exact “singleton-run command economy” direction, I expect a high chance that those concepts matter.
Zero to One is a book that everyone knows about, but somehow it’s still underrated.
Indefinite v definite in particular is a frame that’s stuck with me.
Indefinite:
finance
consulting
“moving upwind” / “keeping your options open”
investing in diversified index funds
theories of impact based on measuring, forecasting, and understanding
Definite:
entrepreneurship
engineering
the Apollo Program
commitment
theories of impact based on specific solution plans and leverage points
I think I agree with all of this.
(Except maybe I’d emphasise the command economy possibility slightly less. And compared to what I understand of your ranking, I’d rank competition between different AGIs/AGI-using factions as a relatively more important factor in determining what happens, and values put into AGIs as a relatively less important factor. I think these are both downstream of you expecting slightly-to-somewhat more singleton-like scenarios than I do?)
EDIT: see here for more detail on my take on Daniel’s takes.
Overall, I’d emphasize as the main point in my post: AI-caused shifts in the incentives/leverage of human v non-human factors of production, and this mattering because the interests of power will become less aligned with humans while simultaneously power becomes more entrenched and effective. I’m not really interested in whether someone should save or not for AGI. I think starting off with “money won’t matter post-AGI” was probably a confusing and misleading move on my part.
the strategy of saving money in order to spend it after AGI is a bad strategy.
This seems very reasonable and likely correct (though not obvious) to me. I especially like your point about there being lots of competition in the “save it” strategy because it happens by default. Also note that my post explicitly encourages individuals to do ambitious things pre-AGI, rather than focus on safe capital accumulation.
Shapley value is not that kind of Solution. Coherent agents can have notions of fairness outside of these constraints. You can only prove that for a specific set of (mostly natural) constraints, Shapeley value is the only solution. But there’s no dutchbooking for notions of fairness.
I was talking more about “dumb” in the sense of violates the “common-sense” axioms that were earlier established (in this case including order invariance by assumption), not “dumb” in the dutchbookable sense, but I think elsewhere I use “dumb” as a stand-in for dutchbookable so fair point.
See https://www.lesswrong.com/posts/TXbFFYpNWDmEmHevp/how-to-give-in-to-threats-without-incentivizing-them for the algorithm when the payoff is known.
Looks interesting, haven’t had a chance to dig into yet though!
Something that I feel is missing from this review is the amount of intuitions about how minds work and optimization that are dumped at the reader. There are multiple levels at which much of what’s happening to the characters is entirely about AI. Fiction allows to communicate models; and many readers successfully get an intuition for corrigibility before they read the corrigibility tag, or grok why optimizing for nice readable thoughts optimizes against interpretability.
I think an important part of planecrash isn’t in its lectures but in It’s story and the experiences of its characters. While Yudkowsky jokes about LeCun refusing to read it, it is actually arguably one of the most comprehensive ways to learn about decision theory, with many of the lessons taught through experiences of characters and not through lectures.
Yeah I think this is very true, and I agree it’s a good way to communicate your worldview.
I do think there are some ways in which the worlds Yudkowksy writes about are ones where his worldview wins. The Planecrash god setup, for example, is quite fine-tuned to make FDT and corrigibility important. This is almost tautological, since as a writer you can hardly do anything else than write the world as you think it works. But it still means that “works in this fictional world” doesn’t transfer as much to “works in the real world”, even when the fictional stuff is very coherent and well-argued.
It doesn’t contain anything I would consider a spoiler.
If you’re extra scrupulous, the closest things are:
A description of a bunch of stuff that happens very early on to set up the plot
One revelation about the character development arc of a non-major character
A high-level overview of technical topics covered, and commentary on the general Yudkowskian position on them (with links to precise Planecrash parts covering them), but not spoiling any puzzles or anything that’s surprising if you’ve read a lot of other Yudkowsky
A bunch of long quotes about dath ilani governance structures (but these are not plot relevant to Planecrash at all)
A few verbatim quotes from characters, which I guess would technically let you infer the characters don’t die until they’ve said those words?
Important other types of capital, as the term is used here, include:
the physical nuclear power plants
the physical nuts and bolts
data centres
military robots
Capital is not just money!
Why would an AI want to transfer resources to someone just because they have some fiat currency?
Because humans and other AIs will accept fiat currency as an input and give you valuable things as an output.
Surely they have some better way of coordinating exchanges.
All the infra for fiat currency exists; I don’t see why the AIs would need to reinvent that, unless they’re hiding from human government oversight or breaking some capacity constraint in the financial system, in which case they can just use crypto instead.
It’s possible that it instead ends up belonging to whatever AI has the best military robots.
Military robots are yet another type of capital! Note that if it were human soldiers, there would be much more human leverage in the situation, because at least some humans would need to agree to do the soldering, and presumably would get benefits for doing so, and would use the power and leverage they accrue from doing so to push broadly human goals.
The recruitment company is toast.
Or then the recruitment company pivots to using human labour to improve AI, as actually happened with the hottest recent recruiting company! If AI is the best investment, then humans and AIs alike will spend their efforts on AI, and the economy will gradually cater more and more to AI needs over human needs. See Andrew Critch’s post here, for example. Or my story here.
The scenario does not say that AI progress slows down. What I imagined to be happening is that after 2028 or so, there is AI research being done by AIs at unprecedented speeds, and this drives raw intelligence forward more and more, but (1) the AIs still need to run expensive experiments to make progress sometimes, and (2) basically nothing is bottlenecked by raw intelligence anymore so you don’t really notice it getting even better.