Please be more specific about what it is that you’re warning against, and why. You could be talking about belief in qualia or quantum mind theory, belief in psychic powers or secret aliens, AI psychosis or belief in AI sentience… Personally I have a particular interest in what you could call anti-woo with respect to consciousness, philosophies like eliminativism and illusionism and even functionalism which want to deny its reality and/or claim that it is nothing but illusion, words, function, etc. At the fringes, some of these philosophies motivated by materialism, turn into the “Platonic materialism” of Moravec, Tegmark, et al, according to which the world in general is a mathematical object, an equivalence class of computer programs, and so on. These ideas are interesting but they also have a mystical aspect, it’s just a mysticism that is appealing for mathematically inclined people. It’s a place where arguably anti-woo turns into a form of woo… For that matter, there are ideas which have a lot of currency in rationalist-adjacent spaces, like acausal interactions, quantum immortality, the simulation hypothesis, which might also be regarded as woo… So, please spell out in more detail what kind of thoughts you think we should avoid, and why.
Mitchell_Porter
Why not?
I think that a sane civilization tries really hard to both solve alignment and align AI to, specifically, the CEV of All The People Everywhere
Why should aligning with lots of people or all people, turn out better than aligning just with one? I don’t recall seeing an overt argument for this, so here are the reasons as I intuit them. The reasons seem to be (1) a hope that the desires of average people will thereby drown out the desires of megalomaniacs and other undesirables, by being more numerous (2) a hope that trying to aggregate the desires of everyone will result in a system that prevents any one person from imposing on others, as a side effect of trying to give everyone some of what they want.
So if I have grasped your own background, you come from an American social layer which is itself pretty elite in terms of social and cultural capital. You were socially adjacent to many people who did carry out the multi-year performative process, but you declined to do so, and the system did not reach out to take you on board anyway, and this is part of your evidence that it is something less than a perfect meritocracy.
I am most reminded of Eric Weinstein, who went to Harvard, but didn’t make it into the patronage networks-within-the-network, and 40 years later now specializes in elaborating his own version of how the system is deficient, corrupt, etc., while having made his own living among the counter-elite of finance, and in particular ending up working for Peter Thiel.
I don’t know your story at all, all I know is that you can write some good essays (e.g. the recent “Compradorization”), but I haven’t tried to find out what social and economic matrix you personally inhabit.
As for the general theme of alleged elites that aren’t actually the best, I have no trouble at all believing that. I would support a Swiss-cheese model of deficiencies in the world’s institutions and cultures, that leads to lost opportunities at every level, from people who are close to the very top of the pyramid but get passed over in favor of the second best, down to gifted individuals who nonetheless end up homeless, dead, or mad, because they can’t get the most elementary kind of social leverage.
My own situation is something like this: I’m Australian, I grew up without any connection to the professional classes or the middle class, but got as far as being a reserve for our IMO team, started a university degree, discovered that the human race in general was resigned to everyone spending their lives working for money and then dying of natural causes, found the Internet as a space where alternative ideas of all kinds had a chance, and over the long term spent my life below the poverty line but never far from academia, and educating myself and pursuing projects online.
I have accomplished a few things, but only a small fraction of what I might expect to have accomplished, if I had found or created a niche at a higher economic level. So I have a sense of profound ongoing lost opportunities, and I don’t just mean a loss for myself—there are works that I could have produced decades ago, which probably would have been in the canons of this or that intellectual subculture if I had actually been able to finish them, judging by what’s there now. Of course I have spent a bit of time thinking about this. If I had been the kind of person who knew at an earlier age how the game is played, perhaps my output would have been mediocre, rather than being those counterfactual works of excellence. Or maybe that’s just a cope.
As you mention, Australia has its own attempt at a meritocratic system, one that I don’t know much about (e.g. I have never heard of those Victorian high schools that you mention). I’m sure it has its deficiencies but I always felt that the barriers I personally faced were of a more fundamental nature, given the nature of my interests and ambitions. To mention one basic recurring example, if the average human psyche had the wisdom and the willpower to see a cure for ageing as something worth trying for, that would have been part of mainstream culture, institutions, and agenda decades ago, if not centuries. The most sophisticated explanation for such resistance that I ever found is in the works of Celia Green, who basically argues that humans learn to repress their ambitions—whether of that kind, or much lesser—for the sake of avoiding the difficulty and pain of failure, and then they take out this repressed frustration on each other in various ways.
In any case, such barriers can be overcome, but that requires persuasive power, political instincts, the ability to call upon other forms of credibility, and so forth, combined with the original insight that things could and should be different.
Returning to your own essay, as you might imagine, I am much more familiar with overlooked talent and lost opportunities that are right out on the social margins, rather than face to face with the central struggles for status and power. If you go far enough outside the system, you find rival elites, counter-elites, the makings of new future elites, as well as lost civilizations, futures that could have been, defeated and forgotten counter-elites of the past. And changes do occur, on small and large scales. The tech right obviously considers itself a new meritocracy that can use the turbulence of Trumpist populism to kick over the moribund shells of liberal America and replace them with AI-powered techno-optimist adhocracy, or whatever the new order is envisaged to be.
As a believer in short AI timelines, many of these discussions seem too late to me. The basic transition we face is not reform of meritocracy, or a changeover of national or international elites, but the replacement of the human race itself by something else. But they can still tell us something about how the human world worked, even as it prepares to leave the historical stage.
I’m going to regret this, but: if I ask my intuition for old-school anime avatars for the other western AI leaders, it comes back with: Mega Man for Elon Musk, Doraemon for Dario Amodei, and Dr Slump for Demis Hassabis.
This is the first I have heard of Unslop, and I would be interested, but the page at hyperstition.ai is unreadable for me. Is the deadline to submit already past?
edit: I found a web-form URL in the source of the page and submitted an application.
It’s also my understanding that Anthropic has historically been relatively leak-free, until a certain memo leaked a few weeks ago during the DoW incident. Supposedly twice is still coincidence, not enemy action, but it does feel like a questionable coincidence and I wonder if the same person is responsible for both leaks.
I agree that it’s suspicious as hell. This one also happened while Dario Amodei was outside the US, doing a deal with the Australian government.
What kind of algorithm and what kind of training set could fit on a laptop and still produce superintelligence? It sounds like the algorithm or its designer might need to be superintelligent already for that to happen.
Some additional comments:
I would expect “multipolar” bodies like the Shanghai Cooperation Organization and the BRICS association to play a role in sorting out a new economic and security regime, in the aftermath of an American defeat. These groupings are explicitly designed for such purposes, and there are Middle Eastern oil producers among their members.
The USA can leave the region, but Israel is stuck there, and Israel and Iran are mortal enemies. This year’s war against Iran, last year’s “12 day” war against Iran, the war in Gaza, and the war in Lebanon are all part of the one conflict. Maybe this is where Europe would come into play.
We should actually ask, not just “what if the US loses”, but “what if the US wins”. Of course, the meaning of victory has turned out to be elusive. The maximum form of victory would be a regime change that resulted in a secular pro-western democracy, and I think this would revive the idea of spreading liberal democracy throughout Eurasia, even to Russia and China.
I suppose a lesser form of American victory would be if there’s no regime change in Iran, but the strait reopens. That seems like a very unstable situation, because the USA would probably want to rebuild its bases in the Arab countries of the Gulf, but would Iran sit still for that? Iran’s actions may seem extremely aggressive—closing the strait, attacking numerous neighboring countries—but their ideological preference is to see the US (and Israel) out of the region entirely, so it makes sense that, at a time when the US said it was aiming for regime change, Iran would in response go for broke.
The main precedent we have for coexistence between Iran and US forces is the Obama nuclear deal, which was part of a new strategic concept in which, rather than treat Israel as a no-limits ally, the US would treat both Israel and Iran as part of a regional balance of power to be managed by America. Israel regarded this as intolerable and immediately used its influence within the US to campaign against the deal, and the next year Trump was elected for the first time.
Another paradigm of Iran-US coexistence, that seems rather arcane now, is the second half of last year, after the 12-day war, and its current spokesperson is Joe Kent, the former counterterrorism czar who endorsed 2025′s war as having established deterrence, but rejected 2026′s war as reflecting Israeli interests rather than American interests. I suppose you could say the world is now looking for a third paradigm of US-Iran coexistence, but as I argued above, it’s unstable because of Israel’s fear of Iranian reconstruction and Iran’s fear of American base reconstruction.
If you want to see advocates of total American and Iranian victory, respectively, I suggest the Youtube streamer Goldie Ghamari (a supporter of secular constitutional monarchy in Iran) and the Sri Lankan essayist Indi Samarajiva (an anti-imperialist who has embraced Iran’s leadership).
Previous comments by me on this topic: guesstimating the odds of various outcomes, a week ago; how this could intersect with the rise of AI.
edit: One more thought. Ghamari recently tweeted that Mohammed-Bagher Ghalibaf—the Iranian parliamentary speaker who seems to have replaced Ali Larijani as the man at the crossroads of Iranian power—“is the new Gorbachev”. It suggests a short-term scenario in which US Marines are keeping the strait open—perhaps by occupying Kharg Island? - while Ghalibaf is trying to keep the Iranian system going, a problem since he will have to deal both with IRGC militants who want to fight Israel and bring the Mahdi, and a populace more angry and alienated than ever after the January massacre of protesters overseen by Larijani.
There is a lot of truth to this sociology of philosophy that you present, although if I were telling the tale, I would have a different emphasis—namely, that intellectual institutions are capable of canonizing some dogma as truth when it’s not known to be true at all. In other words, my version of the tale would be about dogma, delusion, and rationalization, and I would take care to emphasize that these recurring philosophical questions have not actually been answered decisively and correctly, it’s just that cliques and cultures tend to settle on specific possible answers and then dogmatically defend them.
AI 2027 versus World War 2027
There’s a thing people in AI safety leave unspoken: if we do align AI successfully (far from a given), we still have the problem of who it’s aligned to.
My ideas about alignment derive from the dark ages when we talked about “Friendly AI”, and I do not keep up with today’s “AI safety” literature in any systematic way.
But may I point out that today’s literature makes a basic distinction between “intent alignment” and “values alignment”. An “intent-aligned” AI is really good at discerning what its user wants and fulfilling that; whereas a “values-aligned” AI makes its decisions on the basis of values, not just scrupulous obedience.
This can still be mapped onto your dichotomy between “leviathan” and “the people”, e.g. if you can identify different class interests distinct enough to become different value systems. But in general, if your proposal is to align AI with “the people”, you face the problem that the people are actually a mass of individuals with distinct and often contradictory values.
I think within the field of AI safety, there is in fact considerable awareness of the problem of AI-enhanced abuse of power. AI dictatorship is a long-known form of “s-risk”, though lately people have talked more about the CEOs of AI companies becoming the absolute rulers of the world, and it’s only as America and China incorporate AI into their militaries and governments, that people again begin to talk about governmental dictatorship.
However, we do not know how stable AI-enhanced human dictatorship is, compared to outright takeover by AIs themselves. Delegate everything to intelligent machines, and there’s a good chance that some purpose native to the machines will emerge and overwhelm the diktats of the human dictator.
So a lot of people will agree with you that AI needs to be aligned to “norms” rather than to individuals. But, what norms, which norms?
By the way, note that the AI faction allied with Trump 2.0 policy is overtly against any form of universal values alignment, because they consider that to be dictatorship. e/acc co-founder Gill Verdon seems to have the most thought-out version of this, when he argues that AI civilization as a whole needs to orient itself around maximizing energy use (and will do so darwinistically). The argument is that values are complex and constantly shift anyway, whereas material prosperity follows energy abundance and is the more objective indicator of progress.
The “ASI-pilled” part of society is mostly a subset of (1) people working with computers (2) people who read or watch science fiction (3) people who concern themselves with the big picture. LW rationalism is just a sub-subset of that.
Consider Musk, Altman, Amodei, Hassabis. They have all said it’s coming. Are they part of the rationalist circle? Not really. They know about us, they may agree in some areas, but they’ll disagree in others and their personal philosophical and social networks are not centered here. The same would apply to most of their employees, to various intellectuals and public figures who have said it’s coming, all the way down to the scattered private individuals who picked up the idea from who knows where.
Search X and Reddit for conversations about ASI, and you should find people talking about it who have no connection to this place (or even have a negative view of LW’s doomer take on ASI).
I suggest contacting the International Society of Solipsists, they may have some relevant organizational experience…
On a private forum, I just listed possible outcomes and probabilities as:
US leaves Persian Gulf and Eurasian powers work with Iran to establish new economic and security order there, 10%
Islamic Republic falls and is followed by new Eurasian democracy crusade ultimately aimed at Russia and China, 60%
Regime falls but no broader democracy crusade OR regime stays but strait is reopened, 30% combined
These are political scenarios and you’re asking about economics. My intuition is that there could be a world recession but not a world depression.
When did everything go wrong, exactly?
I have said for some time that the problem is much deeper. The human race in general was never on board with transhumanism. The idea of radical life extension has been around for millennia, it has been scientifically plausible for decades if not centuries, but it has always been a marginal concern. There was never a society which organized to make the cure of ageing a major priority.
There has been an incremental improvement over time, both in medical capability (thanks to the progress of conventional medicine) and in openness to life extension (partly thanks to science fiction, perhaps). But it’s almost as if humanity backed its way into this improved situation, under the pressure of immediate concerns (e.g. specific illnesses, individual grief), without ever having consciously adopted a futurist vision like those you describe. At the level of individual psychology, and even more at the level of mass psychology, most people are completely resigned to living out the historically normal human life cycle.
Nonetheless, we actually have a form of transhumanism in power now, but it’s this AI-centric version, half of whose protagonists are in denial about what they are creating. Many of the others think they can skip biology entirely, and just go straight to mind uploading or creation of benevolent AI, or even believe they are in a simulation. This points to a divide within transhumanism itself (and adjacent movements). But socially and politically, I think denial of the full implications of AI, is the main enabling factor. There is no politician who runs for office on the platform of creating non-biological superhuman intelligence. It’s only the tech CEOs who talk directly about anything like that.
I have been contemplating a post about different forms of transhumanism which would go into more detail about all this.
From a very broad perspective, not even focused on Earth, but just on the possible destinies of intelligent life in the cosmos once technology comes into play… It would not be surprising to know that in the encounter with technology, intelligent species often blow up their world or inadvertently replace themselves with a successor species, and only sometimes manage to preserve their own existence and imperatives. It’s just that we also get to live through one instance of such an encounter in person.
Does anyone know how June Ku is doing? Website is down and no tweets for two years.
Having the right simple idea (Darwin, Turing) can be the kernel of everything. The correct ontology could be something like, the correct understanding of entanglement in quantum gravity, plus a precisely stated panprotopsychism that implies consciousness at the human level. The correct meta-ethics may be one of the known proposals (example by a CEV theorist), grounded in the correct ontology’s account of intentionality.
I mention these concrete proposals, not out of commitment to their correctness, but as examples of what the kernel of an answer could look like.
Last month, I posted a “research agenda for the final year”. This was an attempt to rise to the challenge of our situation: AI is advanced enough now, that it would be unsurprising if it reached superintelligence this year. My expectation is that superintelligence rapidly leads to AI takeover of the world and the end of human sovereignty. If all that were true, then we only have months left in which to elevate the theory and practice of alignment to the level needed for superintelligence.
The next step for me is to emphasize that I am available for paid work in this area. That research agenda outlines how I think about the issues; so in principle, I could be hired to work exactly on that agenda, or on some subset of it. But in practice I have considerable flexibility. My main practical constraint is that I am looking for remote work that will allow me to move from Australia, where I am now, to Canada, for family reasons. If this really is the end, that’s where I ought to be. And if we do still have a few more years, being in North America should be a step up in my ability to make a difference.
Sabine Hossenfelder just promoted a paper from last year in which quantum states of chaotically oscillating gravitational collapse are found to encode Riemann-zeta-like functions. This “BKL” oscillation also occurs in Tipler’s Omega Point cosmology, where it’s supposed to be the energetic pump that drives an infinite amount of information processing in finite physical time prior to the Big Crunch singularity. Maybe it’s time to revive Tipler’s idea as a multi-multiverse cosmology in which each ensemble of possible histories ends with a pseudorandom number-theory object encoding a distinct superintelligent God. :-)
(Please consider this a slightly misaligned April Fool post.)