My government name is Mack Gallagher. Crocker’s Rules. I am an “underfunded” “alignment” “researcher”. DM me if you’d like to fund my posts, or my project.
I post some of my less-varnished opinions on my personal blog. In the past they went on my Substack.
If you like arguing with me on LessWrong, at present I’m basically free round the clock to continue interesting arguments in my Discord.
The Aisafety[.]info group has collated some very helpful maps of “who is doing what” in AI safety, including this recent spreadsheet account of technical alignment actors and their problem domains / approaches as of 2024 [they also have an AI policy map, on the off chance you would be interested in that].
I expect “who is working on inner alignment?” to be a highly contested category boundary, so I would encourage you not to take my word for it, and to look through the spreadsheet and probably the collation post yourself [the post contains possibly-clueful-for-your-purposes short descriptions of what each group is working on], and make your own call as to who does and doesn’t [likely] meet your criteria.
But for my own part, it looks to me like the major actors currently working on inner alignment are the Alignment Research Center [ARC] and Orthogonal.
You probably can’t beat reading old MIRI papers and the Arbital AI alignment page. It’s “outdated”, but it hasn’t actually been definitively improved on.
Yes, I think that’s a validly equivalent and more general classification. Although I’d reflect that “survive but lack the power or will to run lots of ancestor-simulations” didn’t seem like a plausible-enough future to promote it to consideration, back in the ’00s.
But, [“I want my father to accept me”] is already a very high level goal and I have no idea how it could be encoded in my DNA. Thus maybe even this is somehow learned. [ . . . ] But, to learn something, there needs to be a capability to learn it—an even simpler pattern which recognizes “I want to please my parents” as a refined version of itself. What could that proto-rule, the seed which can be encoded in DNA, look like? [ . . . ] equip organisms with something like “fundamental uncertainty about the future, existence and food paired with an ability to recognize if probability of safety increases”
Humans [and animals in general] are adaptation-executers, not fitness-maximizers [ obligatory link, although my intention is not particularly to direct you to [re-]read ].
In the extreme limit, if natural [/sexual] selection could just compose us of the abstract principle “survive and reproduce”, it would. But natural [/sexual] selection is a blind idiot god [obligatory link] without the capacity for such intelligent design; its entire career consists of resorting to crude hacks, precisely of the mold “directly wire in an instinctual compulsion for the child to please its father”. I forcefully recommend Byrnes’s writing on imprinting in animals and humans; it changed my mind about this some years ago.
“Why am I aligned with my own [stated] values [as opposed to my less-prosocial, subconscious urges]?”
and
“Why am I aligned with evolution’s values [to the extent that I am]?”
are different questions.
[ Note: I strongly agree with some parts of jbash’s answer, and strongly disagree with other parts. ]
As I understand it, Bostrom’s original argument, the one that got traction for being an actually-clever and thought-provoking discursive fork, goes as follows:
Future humans in specific, will at least one of: [ die off early, run lots of high-fidelity simulations of our universe’s history [“ancestor-simulations”], decide not to run such simulations ].
If future humans run lots of high-fidelity ancestor-simulations, then most people who subjectively experience themselves as humans living early in a veridical human history, will in fact be living in non-base-reality simulations of such realities, run by posthumans.
If one grants that our ancestors are likely to a] survive, and b] not elect to run vast numbers of ancestor-simulations [ both of which assumptions felt fairly reasonable back in the ’00s, before AI doom and the breakdown of societal coordination became such nearly felt prospects ], then we are forced to conclude that we are more likely than not living in one such ancestor-simulation, run by future humans.
It’s a valid and neat argument which breaks reality down into a few mutually-exclusive possibilities—all of which feel narratively strange—and forces you to pick your poison.
Since then, Bostrom and others have overextended, confused, and twisted this argument, in unwise attempts to turn it into some kind of all-encompassing anthropic theory. [ I Tweeted about this over the summer. ]
The valid, original version of the Simulation Hypothesis argument relies on the [plausible-seeming!] assumption that posthumans, in particular, will share our human interest in our species’ history, in particular, and our penchant for mad science. As soon as your domain of discourse extends outside the class boundaries of “future humans”, the Simulation Argument no longer says anything in particular about your anthropic situation. We have no corresponding idea what alien simulators would want, or why they would be interested in us.
Also, despite what Aynonymousprsn123 [and Bostrom!] have implied, the Simulation Hypothesis argument was never actually rooted in any assumptions about local physics. Changing our assumptions about such factors as [e.g.] the spatial infinity of our local universe, quantum decoherence, or a physical Landauer limit, doesn’t have any implications for it. [ Unless you want to argue for a physical Landauer limit so restrictive it’d be infeasible for posthumans to run any ancestor-simulations at all. ]
So, while the Simulation Hypothesis argument can imply you’re being simulated by posthumans, if and only if you strongly believe posthumans will both of [ a] not die, b] not elect against running lots of ancestor-simulations ], it can’t prove you’re being simulated in general. It’s just not that powerful.
Then your neighbor wouldn’t exist and the whole probability experiment wouldn’t happen from their perspective.
From their perspective, no. But the answer to
In which ten billion interval your birth rank could’ve been
changes. If by your next-door neighbor marrying a different man, one member of the other (10B − 1) is thus swapped out, you were born in a different 10B interval.
Unless I’m misunderstanding what you mean by “In which ten billion interval”? What do you mean by “interval”, as opposed to “set [of other humans]”, or just “circumstances”?
If we’re discussing the object-level story of “the breakfast question”, I highly doubt that the results claimed here actually occurred as described, due [as the 4chan user claims] to deficits in prisoner intelligence, and that “it’s possible [these people] lack the language skills to clearly communicate about [counterfactuals]”.
Did you find an actual study, or other corroborating evidence of some kind, or just the greentext?
Is the quantum behavior itself, with the state of the system extant but stored in [what we perceive as] an unusual format, deterministic? If you grant that there’s no in-the-territory uncertainty with mere quantum mechanics, I bet I can construct a workaround for fusing classical gravity with it that doesn’t involve randomness, which you’ll accept as just as plausible as the one that does.
To me the more interesting thing is not the mechanism you must invent to protect Heisenberg uncertainty from this hypothetical classical gravitational field, but the reason you must invent it. What, in Heisenberg uncertainty, are you protecting?
Does standard QED, by itself, contain something of the in-the-territory state-of-omitted-knowledge class you imagine precludes anthropic thinking? If not, what does it contain, that requires such in-the-territory obscurity to preserve its nature when it comes into contact with a force field that is deterministic?
[ Broadly agreed about the breakfast hypothetical. Thanks for clarifying. ]
In the domain of anthropics reasoning, the questions we’re asking aren’t of the form
A) I’ve thrown a six sided die, even though I could’ve thrown a 20 sided one, what is the probability to observe 6?
or
B) I’ve thrown a six sided die, what would be the probability to observe 6, if I’ve thrown a 20 sided die instead?
In artificial spherical-cow anthropics thought experiments [like Carlsmith’s], the questions we’re asking are closer to the form of
A six-sided die was thrown with 60% probability; with probability 40%, a 20-sided die was thrown. A six was observed. Now what are your posterior probabilities on which die was thrown?
In real-world, object-level anthropic reasoning [like the kind Hanson is doing], the questions we’re asking are of the form
An unknown number of n-sided die were thrown, and landed according to unknown metaphysics to produce the reality observed, which locally works according to known deterministic physics, but contains reflective reasoners able to generate internally coherent counterfactuals which include apparently plausible violations of “what would happen according to physics alone”. Attempt to draw any conclusions about metaphysics.
You say:
We know very well the size of set of possible outcomes for “In which ten billion interval your birth rank could’ve been”. This size is 1. No amount of pregnancy complications could postpone or hurry your birth so that you managed to be in a different 10 billion group.
What if my next-door neighbor’s mother had married and settled down with a different man?
I enjoyed reading this post; thank you for writing it. LessWrong has an allergy to basically every category Marx is a member of—“armchair” philosophers, socialist theorists, pop humanities idols—in my view, all entirely unjustified.
I had no idea Marx’s forecast of utopia was explicitly based on extrapolating the gains from automation; I take your word for it somewhat, but from being passingly familiar with his work, I have a hunch you may be overselling his naivete.
Unfortunately, since the main psychological barrier to humans solving the technical alignment problem at present is not altruistic intentions, but raw cognitive intelligence, any meta-alignment scheme that proposes to succeed today has far more work cut out for it than just ensuring AGI-builders are accounting for risk to the best of their ability. It has to make the best of their ability good enough. That involves, at the very minimum, an intensive selection program for geniuses who are then placed in a carefully incentives-aligned research environment, and probably human intelligence enhancement.
Are you familiar with Yudkowsky’s/Miles’s/Christiano’s AI Corrigibility concept?
So yes, I think this is a valid lesson that we can take from Principle (A) and apply to AGIs, in order to extract an important insight. This is an insight that not everyone gets, not even (actually, especially not) most professional economists, because most professional economists are trained to lump in AGIs with hammers, in the category of “capital”, which implicitly entails “things that the world needs only a certain amount of, with diminishing returns”.
This trilemma might be a good way to force people-stuck-in-a-frame-of-traditional-economics to actually think about strong AI. I wouldn’t know; I honestly haven’t spent a ton of time talking to such people.
Principle [A] doesn’t just say AIs won’t run out of productive things to do; it makes a prediction about how this will affect prices in a market. It’s true that superintelligent AI won’t run out of productive things to do, but it will also change the situation such that the prices in the existing economy won’t be affected by this in the normal way prices are affected by “human participants in the market won’t run out of productive things to do”. Maybe there will be some kind of legible market internal to the AI’s thinking, or [less likely, but conceivable] a multi-ASI equilibrium with mutually legible market prices. But what reason would a strongly superintelligent AI have to continue trading with humans very long, in a market that puts human-legible prices on things? Even in Hanson’s Age of Em, humans who choose to remain in their meatsuits are entirely frozen out of the real [emulation] economy, very quickly in subjective time, and to me this is an obvious consequence of every agent in the market simply thinking and working way faster than you.
Wild ahead-of-of-time guess: the true theory of gravity explaining why galaxies appear to rotate too slowly for a square root force law will also uniquely explain the maximum observed size of celestial bodies, the flatness of orbits, and the shape of galaxies.
Epistemic status: I don’t really have any idea how to do this, but here I am.
When I stated Principle (A) at the top of the post, I was stating it as a traditional principle of economics. I wrote: “Traditional economics thinking has two strong principles, each based on abundant historical data”,
I don’t think you think Principle [A] must hold, but I do think you think it’s in question. I’m saying that, rather than taking this very broad general principle of historical economic good sense, and giving very broad arguments for why it might or might not hold post-AGI, we can start reasoning about superintelligent manufacturing [including R&D] and the effects it will have, more granularly, out the gates.
Like, with respect to Principle [C] my perspective is just “well of course the historical precedent against extremely fast economic growth doesn’t hold after the Singularity, that’s more or less what the Singularity is”.
Edit: Your rewrite of Principle [B] did make it clear to me that you’re considering timelines that are at least somewhat bad for humans; thank you for the clarification. [Of course I happen to think we can also discard “AI will be like a manufactured good, in terms of its effects on future prices”, out the gates, but it’s way clearer to me now that the trilemma is doing work on idea-space.]
You seem to have misunderstood my text. I was stating that something is a consequence of Principle (A),
My position is that if you accept certain arguments made about really smart AIs in “The Sun is Big”, Principle A, by itself, ceases to make sense in this context.
costs will go down. You can argue that prices will equilibriate to costs, but it does need an argument.
Assuming constant demand for a simple input, sure, you can predict the price of that input based on cost alone. The extent to which “the price of compute will go down”, is rolled in to how much “the cost of compute will go down”. But IIUC, you’re more interested in predicting the price of less abstract assets. Innovation in chip technology is more than just making more and more of the same product at a lower cost. [ “There is no ‘lump of chip’.” ] A 2024 chip is not just [roughly] 2^10 2004 chips—it has logistical advantages, if nothing else. And those aren’t accounted for if you insist on predicting chip price using only compute cost and value trendlines. Similar arguments hold for all other classes of material technological assets whose value increases in response to innovation.
“AI will [roughly] amount to X”, for any X, including “high-skilled entrepreneurial human labor” is a positive claim, not a default background assumption of discourse, and in my reckoning, that particular one is unjustified.
Thank you for writing this and hopefully contributing some clarity to what has been a confused area of discussion.
So here’s a question: When we have AGI, what happens to the price of chips, electricity, and teleoperated robots?
(…Assuming free markets, and rule of law, and AGI not taking over and wiping out humanity, and so on. I think those are highly dubious assumptions, but let’s not get into that here!)
Principle (A) has an answer to this question. It says: prices equilibrate to marginal value, which will stay high, because AGI amounts to ambitious entrepreneurial skilled labor, and ambitious entrepreneurial skilled labor will always find more new high-value things to do. That, incidentally, implies that human labor will retain a well-paying niche—just as less-skilled labor today can still get jobs despite more-skilled labor also existing.
First off, I’m guessing you’re familiar with the economic arguments in The Sun is Big.
Secondly -
If we’re talking about prices for the same chips, [rate of] electricity, teleoperated robots, etc., of course they’ll go down, as the AGI will have invented better versions.
AGI amounts to ambitious entrepreneurial skilled labor
This is really just a false thing to believe about AGI, from us humans’ perspective. It amounts to a new world political order. Unless you specifically build it to prevent all other future creations of humanity from becoming politically interventionist superintelligences, while also not being politically interventionist itself.
I didn’t say he wasn’t overrated. I said he was capable of physics.
Did you read the linked post? Bohm, Aharonov, and Bell misunderstood EPR. Bohm’s and Aharonov’s formulation of the thought experiment is easier to “solve” but does not actually address EPR’s concern, which is that mutual non-commutation of x-, y-, and z-spin implies hidden variables must not be superfluous. Again, EPR were fine with mutual non-commutation, and fine with entanglement. What they were pointing out was that the two postulates don’t make sense in each other’s presence.
Your linked post on The Obliqueness Thesis is curious. You conclude thus:
Obliqueness obviously leaves open the question of just how oblique. It’s hard to even formulate a quantitative question here. I’d very intuitively and roughly guess that intelligence and values are 3 degrees off (that is, almost diagonal), but it’s unclear what question I am even guessing the answer to. I’ll leave formulating and answering the question as an open problem.
I agree, values and beliefs are oblique. The 3 spatial dimensions are also mutually oblique, as per General Relativity. A theory of obliqueness is meaningless if it cannot specify the angles [ I think in a correct general linear algebra, everything would be treated as [at least potentially] oblique to everything else, but that doesn’t mean I refuse to ever treat the 3 spatial dimensions as mutually orthogonal ].
As with the 3 spatial dimensions in practical ballistics, with the value dimension and the belief dimension in practical AI alignment, there are domains of discussion where it is appropriate to account for the skew between the dimensions and domains where it is appropriate to simply treat them as orthogonal. Discussions of alignment theory such as the ones in which you seek to insert your Obliqueness thesis, are a domain in which the orthogonality assumption is appropriate. We cannot guess at the skew with any confidence in particular cases, and with respect to any particular pre-chosen utility/valence function term versus any particular belief-state [e.g. “what is the dollar value of a tulip?” • “is there a teapot circling Mars?”], the level of skew is almost certain to be negligible.
Planning for anthropically selected futures, on the other hand, is a domain where the skew between values and beliefs becomes relevant. There is less point reasoning in detail about pessimized futures or others that disagree with our values [such as the ones in which we are dead], no matter how likely or unlikely they might be “in a vacuum”, if we’re trying to hyperstition our way into futures we like. But this is an esoteric and controversial argument and not actually required to justify why I don’t think it’s useful to consider [sufficiently] strong AI as “what can eat the sun”.
All that’s required to justify why I don’t think it’s useful to consider [sufficiently] strong AI as “what can eat the sun”, is that what you propose is a benchmark of capability, or intelligence. Benchmarks of intelligence [say, of bureaucrats or chimps] are not questions of fact. They are social fictions chosen for their usefulness. If, in the vast, vast supermajority of the worlds where the benchmark would otherwise be useful—in this case, the worlds where people deploy an AI that they do not know, ahead of time, if it will or will not be strong enough to eat the Sun—it will not be useful for contingent reasons—in this case, because we are all dead—then it is not, particularly, a benchmark we should be etching into the wood, from our present standpoint.
The characters don’t live in a world where sharing or smoothing risk is already seen as a consensus-valuable pursuit; thus, they will have to be convinced by other means.
I gave their world weirdly advanced [from our historical perspective] game theory to make it easier for them to talk about the question.
Yudkowsky’s sequences [/Rationality: AI to Zombies] provide both these things. People did not read Yudkowsky’s sequences and internalize the load-bearing conclusions enough to prevent the current poor state of AI theory discourse, though they could have. If you want your posts to have a net odds-of-humanity’s-survival-improving impact on the public discourse on top of Yudkowsky’s, I would advise that you condense your points and make the applications to concrete corporate actors, social contexts, and Python tools as clear as possible.
[ typo: ‘Merman’ → ‘Mermin’ ]