Bostrom’s argument may be underappreciated. You might like Roman Yampolskiy’s work if you’re deeply interested in exploring the Simulation argument.
Embee
Karma: 61
Can you tell me your p(doom) and AGI timeline? Cause I think we can theoretically settle this:
I give you x$ now and in y years you give me back x times r $ back
Please tell me acceptable y, r for you (ofc in the sense of least-convenient-but-still-profitable)
I think we can conceivably gather data on the combination of “anthropic shadow is real & alignment is hard”.
Predictions would be:
we will survive this
conditional on us finding alien civilizations that reached the same technological level, most of them will have been wiped by AI.
2. is my guess as to why there is a Great Filter. More so than Grabby Aliens.
That’s good to know! Best of luck in your project
Feels deep but I don’t get it.
Would you mind elaborating?
ANTHROPIC IMMORTALITY
Are other people here having the feeling of “we actually probably messed up AI alignment but I think we are going to survive for weird anthropic reasons”?
[Sorry if this is terrible formatting, sorry if this is bad etiquette]I think the relevant idea here is the concept of anthropic immortality. It has been alluded to on LW more time than I could count and has even been discussed up explicitly in this context: https://alignmentforum.org/posts/rH9sXupnoR8wSmRe9/ai-safety-via-luck-2
Eliezer wrote somewhat cryptic tweets referencing it recently:
https://x.com/ESYudkowsky/status/1138936939892002816
https://x.com/ESYudkowsky/status/1866627455286648891
But for several weeks I’ve wished there was a definitive place on the internet where it is examined cause I have trouble wrapping my mind around the idea. Its value, theoretical defects, likelihood (even though it seems to break down probability calculation: https://x.com/ESYudkowsky/status/1138938670881239040 )
It doesn’t help that it is related to and/or confused with quantum immortality (QI) which actually shows up on the internet (see in particular: https://www.lesswrong.com/posts/cjK6CTW9DyFAFtKHp/false-vacuum-the-universe-playing-quantum-suicide, https://www.lesswrong.com/posts/hB2CTaxqJAeh5jdfF/quantum-immortality-a-perspective-if-ai-doomers-are-probably), has its own LessWrong entry and a Wikipedia article. It doesn’t help either that QI has become kind of a meme at this point.
If you check the context, EY is making the point that anthropic immortality is distinct from QI: https://x.com/knosciwi/status/1866619917979754593, which maybe a sign people got them mixed up?
I feel like there are multiple people “reinventing the wheel” and describing the concept independently.All this to say:
- maybe someone should compile a broadly accessible entry!
- thinking about doing it myself but I don’t know how valuable it would be (maybe everyone here nodded along to EY tweets and has a clear mind on this topic)
- could the curious coordinate to explore and document the concept together? perhaps we can start a thread to discuss it further
Humbly pinging relevant people, mainly authors from articles I linked to: @avturchin @Jozdien @James_Miller @Halfwit @Vladimir_Nesov
To me Feynman seems to fall quite on the von Neumann side of the spectrum.
Yes, they seem to represent two completely different types of extreme intelligence which is very interesting. I also agree that vN’s ideas are more relevant for the community.
Yes. Grothendieck is undoubtedly less innovative and curious all across the board.
But I should have mentioned they are not of the same generation. vN helps build the atom bomb while G grows up in a concentration camp.
vN went along a scientific golden age. I’d argue it was probably harder to have the same impact on Science in the 1960s.
I also model G as having disdain for applying mathematical ideas to “impure” subjects. Maybe because of the Manhattan project itself as well as the escalation of the Cold War.
This would be consistent with a whole school of french mathematicians deifying pure math, N. Bourbaki in general, and being generally skeptical of the potential of pure math on the improvement of society, Roger Godement being the stereotype.
My point was that Grothendieck’s mind is interesting to dissect for someone interested in a general theory of intelligence and AI alignment (and that the von Neumann metaphor becomes kinda tiring)
Pet peeve: AI community defaulted to von Neumann as being the ultimate smart human and therefore the basis of all ASI/human intelligence comparison when the mathematician Alexander Grothendieck exists somehow.
Von Neumann arguably had the highest processor-type “horsepower” we know of plus his breadth of intellectual achievements is unparalleled.
But imo Grothendieck is a better comparison point for ASI as his intelligence, while being strangely similar to LLMs in some dimensions, arguably more closely resembles what alien-like intelligence would be:
- solving “impossible” problem through meta-language and abstractions.
- able to think deeply on his own (re-discovered measure theory alone when he was a teenager, re-discovered Poincaré results when undergrad, apparently solved multiple PhD theses in parallel in less than a year)
- almost solely built algebraic geometry (which in turn provided the blueprint for category theory) a domain which scares a part of the mathematics community to this day.
- not your typical child prodigy—
famously bad at computations: “take a prime number. 57 for instance.”
Even from the AI alignment perspective, Grothendieck is fascinating.
Unaligned with “society” incentives and rewards yet having strong moral preferences, in the sense of choosing to work for a public university when he probably could have earned a higher wage elsewhere, holding hardcore communist beliefs, refusing the Fields medal in protest of Soviet Union and on top of that chosing to be stateless.
Disappeared the moment he understood that despite all of that his discoveries were still fueling the industrial-military complex.
Embee’s Shortform
Hi! I’m Embee but you can call me Max.
I’m a mathematics for quantum physics graduate student considering redirecting my focus toward AI alignment research. My background includes:
- Graduate-level mathematics
- Focus on quantum physics
- Programming experience with Python
- Interest in type theory and formal systemsI’m particularly drawn to MIRI-style approaches and interested in:
- Formal verification methods
- Decision theory implementation
- Logical induction
- Mathematical bounds on AI systemsMy current program feels too theoretical and disconnected from urgent needs. I’m looking to:
- Connect with alignment researchers
- Find concrete projects to contribute to
- Apply mathematical rigor to safety problems
- Work on practical implementationsRegarding timelines: I have significant concerns about rapid capability advances, particularly given recent developments (o3). I’m prioritizing work that could contribute meaningfully in a compressed timeframe.
Looking for guidance on:
- Most neglected mathematical approaches to alignment
- Collaboration opportunities
- Where to start contributing effectively
- Balance between theory and implementation
The best pathway towards becoming a member is to produce lots of great AI Alignment content, and to post it to LessWrong and participate in discussions there. The LessWrong/Alignment Forum admins monitor activity on both sites, and if someone consistently contributes to Alignment discussions on LessWrong that get promoted to the Alignment Forum, then it’s quite possible full membership will be offered.
Got it. Thanks.
I’ve noticed that the karma system makes me gravitate towards posts of very high karma. Are there low-karma posts that impacted you? Maybe you think they are underrated or that they fail in interesting ways.
I’m still bothering you with inquiries on user information. I would like to check this in order to write a potential LW post. Do we have data on the prevalence of “mental illnesses” and do we have a rough idea of the average IQ among LWers (or SSCers since the community is adjacent) I’m particulary interested in the prevalence of people with autism and/or schizoid disorders. Thank you very much. Sorry if I used offensive terms. I’m not a native speaker.
What happens if and when a slightly unaligned AGI crowds the forum with its own posts? I mean, how strong is our “are you human?” protection?
Thank you so much.
Does someone have a guesstimate of the ratio of lurkers to posters on lesswrong? With ‘lurker’ defined as someone who has a habit of reading content but never posts stuff (or posts only clarification questions)
In other words, what is the size of the LessWrong community relative to the number of active contributors?
Promising. Where can interested researchers discuss this and what does the question bank look like so far?