Use this thread to (a) upvote topics you’re interested in reading about, (b) agree/disagree with positions, and (c) add new positions for people to vote on.
Note: Hit cmd-f or ctrl-f (whatever normally opens search) to automatically expand all of the poll options below.
Current progress in AI governance will translate with greater than 50% probability into more than a 2 year counterfactual delay of dangerous AI systems
Ambitious mechanistic interpretability is quite unlikely[1] to be able to confidently assess[2] whether AIs[3] are deceptively aligned (or otherwise have dangerous propensities) in the next 10 years.
It is very unlikely AI causes an existential catastrophe (Bostrom or Ord definition) but doesn’t result in human extinction. (That is, non-extinction AI x-risk scenarios are unlikely)
Good AGI-notkilleveryoneism-conscious researchers should in general prioritize working at big AGI labs over working independently, for alignment-focused labs, or for academia marginally more than they currently do.
The ratio of good alignment work done at labs vs independently mostly skews toward labs
Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than independent researchers. Your answer should factor out such considerations.
Someone in the AI safety community (e.g. Yud, Critch, Salamon, you) can currently, within 6 month’s effort, write a 20,000 word document that would pass a threshold for a coordination takeoff on Earth, given that 1 million smart Americans and Europeans would read all of it and intended to try out much of the advice (i.e. the doc succeeds given 1m serious reads, it doesn’t need to cause 1m serious reads). Copy-pasting already-written documents/posts would count.
A basic deontological and straightforward morality (such as that exmplified by Hermione in HPMOR) is basically right; this is in contrast with counterintuitive moralities that suggest evil-tinted people (like Quirrell in HPMOR) are also valid ways of being moral.
The work of agency-adjacent research communities such as artificial life, complexity science and active inference is at least as relevant to AI alignment as LessWrong-style agent foundations research is.
The rationality community will noticeably spill over into other parts of society in the next ten years. Examples: entertainment, politics, media, art, sports, education etc.
At least one American intelligence agency is concerned about the AI safety movement potentially decelerating the American AI industry, against the administration/natsec community’s wishes
Agent foundations research should become more academic on the margin (for example by increasing the paper to blogpost ratio, and by putting more effort into relating new work to existing literature).
Current progress in AI governance will translate with greater than 20% probability into more than a 2 year counterfactual delay of dangerous AI systems
Immersion into phenomena is better for understanding them than trying to think through at the gears-level, on the margin for most people who read LessWrong.
Effective altruism can be well modeled by cynically thinking of it as just another social movement, in the sense that those a part of it are mainly jockeying for in-group status, and making costly demonstrations to their in-group & friends that they care about other sentiences more than others in the in-group. Its just that EA has more cerebral standards than others.
“Open-source LLM-based agent with hacking abilities starts spreading itself over the Internet because some user asked it to do so or to do something like to conquer the world” is a quite probable point-of-no-return regarding AGI risk.
The younger generation of rationalists are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.
At least one of {Anthropic, OpenAI, Deepmind} is net-positive compared to the counterfactual where just before founding the company, its founders were all discretely paid $10B by a time-travelling PauseAI activist not to found the company and to exit the industry for 30 years, and this worked.
The most valuable new people joining AI safety will usually take ~1-3 years of effort to begin to be adequately sorted and acknowledged for their worth, unless they are unusually good at self-promotion e.g. gift of gab, networking experience, and stellar resume.
If rationality took off in China, it would yield higher EV from potentially spreading to the rest of the world than from potentially accelerating China.
When people try to discuss philosophy, math, or science, especially pre-paradigmatic fields such as ai safety, they use a lot of metaphorical thinking to extend from familiar concepts to new concepts. It would be very helpful and people would stop talking past each other so much if they practiced being explicitly aware of these mental representations and directly shared them rather than pretending that something more rigorous is happening. This is part of Alfred Korzybski’s original rationality project, something he called ‘consciousness of abstraction.’
Research into getting a mechanistic understanding of the brain for purposes of at least one of: understanding how values/empathy works in people, brain uploading or improving cryonics/plastination is net positive and currently greatly underfunded.
A Secular Solstice variation designed to work weekly (akin to Sunday Service or Shabbat) would be positive for rationalists, both for community and for the thought processes of the members.
“Intelligence” can be characterized with a similar level of theoretical precision as e.g., heat, motion, and information. (In other words: it’s less like a messy, ad-hoc phenomena and more like a deep, general fact about our world).
You know of a technology that has at least a 10% chance of having a very big novel impact on the world (think the internet or ending malaria) that isn’t included in this list, very similar, or downstream from some element of it: AI, mind uploads, cryonics, human space travel, geo-engineering, gene drives, human intelligence augmentation, anti-aging, cancer cures, regenerative medicine, human genetic engineering, artificial pandemics, nuclear weapons, proper nanotech, very good lie detectors, prediction markets, other mind-altering drugs, cryptocurrency, better batteries, BCIs, nuclear fusion, better nuclear fission, better robots, AR, VR, room-temperature superconductors, quantum computers, polynomial time SAT solvers, cultured meat, solutions to antibiotic resistance, vaccines to some disease, optical computers, artificial wombs, de-extinction and graphene.
Bad options included just in case someone thinks they are good.
Xi Jinping thinks that economic failure in the US or China, e.g. similar to 2008, is one of the most likely things to change the global balance of power.
If we had access to a brain upload (and maybe a world simulator too) we could in principle extract something like a utility function, and the theory behind it relates more to agents in general than it does to humans in particular.
Humans are the dominant species on earth primarily because our individual intelligence surpassed the necessary threshold to sustain civilization and take control of our environment.
If you can write prompt for GPT-2000 such that completion of this prompt results in aligned pivotal act, you can just use knowledge necessary for writing this prompt to Just Build aligned ASI, without necessity to use GPT-2000.
Rationalist rituals like Petrov day or the Secular Solstices should be marginally more emphasized within those collections of people who call themselves rationalists.
Rationality is likely to organically gain popularity in China (e.g. quickly reaching 10,000 people or reaching 100,000 by 2030, e.g. among scientists or engineers, etc).
The ratio of good alignment work done at labs vs in academia mostly skews toward labs
Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than academics. Your answer should factor out such considerations.
The ratio of good alignment work done in academia vs independently mostly skews toward academia
Good meaning something different from impactful here. Possibly AGI labs will pay more attention to academics than independent researchers. Your answer should factor out such considerations.
At least one mole, informant, or spy has been sent by a US government agency or natsec firm to infiltrate the AI safety community by posing as a new member (even if it’s just to ask questions in causal conversations at events about recent happenings or influential people’s priorities).
Most end-to-end “alignment plans” are bad because research will be incremental. For example, Superalignment’s impact will mostly come from adapting to the next ~3 years of AI discoveries and working on relevant subproblems like interp, rather than creating a superhuman alignment researcher.
Conceptual alignment work on concepts like “agency”, “optimization”, “terminal values”, “abstractions”, “boundaries” is mostly intractable at the moment.
Computer science & ML will become lower in relevance/restricted in scope for the purposes of working with silicon-based minds, just as human-neurosurgery specifics are largely but not entirely irrelevant for most civilization-scale questions like economic policy, international relations, foundational research, etc.
Or IOW: Model neuroscience (and to some extent, model psychology) requires more in-depth CS/ML expertise than will the smorgasbord of incoming subfields of model sociology, model macroeconomics, model corporate law, etc.
There is likely a deep compositional structure to be found for alignment, possibly to the extent that AGI alignment could come from “merely” stacking together “microalignment”, even if in non-trivial ways.
If you can’t write a program that produces aligned (under whatever definition of alignment you use) output being run on unphysically large computer, you can’t deduce from training data or weights of superintelligent neural network if it produces aligned output.
There are arguments for convergent instrumental pressures towards catastrophe, but the required assumptions are too strong for the arguments to clearly go through.
Cultural values are something like preferences over pairs of social environments and things we actually care about. So it makes sense to talk about jointly optimizing them.
The younger generation of EAs are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.
The mind (ie. your mind), and how it is experienced from the inside, is potentially a very rich source of insights for keeping AI minds aligned on the inside.
All else equal, a unit of animal suffering should be accorded the same moral weight as an equivalent unit of human suffering. (i.e. equal consideration for equal interests)
Developing a solid human intelligence/skill evaluation metric would be a high-EV project for AI safety, e.g. to make it easier to invest in moving valuable AI safety people to the Bay Area/London from other parts of the US/UK.
MadHatter is an original and funny satirist. The universally serious reaction to his jokeposts is a quintessential example of rationalist humorlessness.
Poll For Topics of Discussion and Disagreement
Use this thread to (a) upvote topics you’re interested in reading about, (b) agree/disagree with positions, and (c) add new positions for people to vote on.
Note: Hit cmd-f or ctrl-f (whatever normally opens search) to automatically expand all of the poll options below.
Prosaic Alignment is currently more important to work on than Agent Foundations work.
LLMs as currently trained run ~0 risk of catastrophic instrumental convergence even if scaled up with 1000x more compute
Academia is sufficiently dysfunctional that if you want to make a great scientific discovery you should basically do it outside of academia.
Pursuing plans that cognitively enhance humans while delaying AGI should be our top strategy for avoiding AGI risk
Current progress in AI governance will translate with greater than 50% probability into more than a 2 year counterfactual delay of dangerous AI systems
Ambitious mechanistic interpretability is quite unlikely[1] to be able to confidently assess[2] whether AIs[3] are deceptively aligned (or otherwise have dangerous propensities) in the next 10 years.
greater than 90% failure
likelihood ratio of 10
I’m refering to which ever AIs are pivotal or cruxy for things to go well prior to human obsolescence.
It is very unlikely AI causes an existential catastrophe (Bostrom or Ord definition) but doesn’t result in human extinction. (That is, non-extinction AI x-risk scenarios are unlikely)
Things will basically be fine regarding job loss and unemployment due to AI in the next several years and those worries are overstated
The current AI x-risk grantmaking ecosystem is bad and could be improved substantially.
People aged 12 to 18 should basically be treated like adults rather than basically treated like children.
It is critically important for US/EU companies to build AGI before Chinese companies.
EAs and rationalists should strongly consider having lots more children than they currently are
Meaningness’s “Geeks Mops and Sociopaths” model is an accurate model of the dynamics that underlie most social movements
Irrefutable evidence of extraterrestrial life would be a good thing.
It was a mistake to increase salaries in the broader EA/Rationality/AI-Alignment ecosystem between 2019 and 2022
Good AGI-notkilleveryoneism-conscious researchers should in general prioritize working at big AGI labs over working independently, for alignment-focused labs, or for academia marginally more than they currently do.
The ratio of good alignment work done at labs vs independently mostly skews toward labs
Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than independent researchers. Your answer should factor out such considerations.
Edit: Also normalize for quantity of researchers.
Someone in the AI safety community (e.g. Yud, Critch, Salamon, you) can currently, within 6 month’s effort, write a 20,000 word document that would pass a threshold for a coordination takeoff on Earth, given that 1 million smart Americans and Europeans would read all of it and intended to try out much of the advice (i.e. the doc succeeds given 1m serious reads, it doesn’t need to cause 1m serious reads). Copy-pasting already-written documents/posts would count.
There is a greater than 20% chance that the Effective Altruism movement has been net negative for the world.
Empirical agent foundations is currently a good idea for a research direction.
A basic deontological and straightforward morality (such as that exmplified by Hermione in HPMOR) is basically right; this is in contrast with counterintuitive moralities that suggest evil-tinted people (like Quirrell in HPMOR) are also valid ways of being moral.
Just like the last 12 months was the time of the chatbots, the next 12 months will be the time of agent-like AI product releases.
The work of agency-adjacent research communities such as artificial life, complexity science and active inference is at least as relevant to AI alignment as LessWrong-style agent foundations research is.
American intelligence agencies consider AI safety to be substantially more worth watching than most social movements
It is possible to make meaningful progress on deceptive alignment using experiments on current models
Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large compute budgets for safety research teams.
Moloch is winning.
“Polyamory-as-a-default-option” would be a better social standard than “Monogamy-as-a-default-option”.
The rationality community will noticeably spill over into other parts of society in the next ten years. Examples: entertainment, politics, media, art, sports, education etc.
At least one American intelligence agency is concerned about the AI safety movement potentially decelerating the American AI industry, against the administration/natsec community’s wishes
I broadly agree with the claim that “most people don’t do anything and the world is very boring”.
On the current margin most people would be better off involving more text-based communication in their lives than in-person communication.
Agent foundations research should become more academic on the margin (for example by increasing the paper to blogpost ratio, and by putting more effort into relating new work to existing literature).
Current progress in AI governance will translate with greater than 20% probability into more than a 2 year counterfactual delay of dangerous AI systems
Rationality should be practiced for Rationality’s sake (rather than for the sake of x-risk).
It is possible to make meaningful progress on ELK using empirical experiments on current models
Language model agents are likely (>20%) to produce AGI (including the generalization to foundation model-based cognitive architectures)
Current AI safety university groups are overall a good idea and helpful, in expectation, for reducing AI existential risk
Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large-scale lobbying efforts in DC.
Immersion into phenomena is better for understanding them than trying to think through at the gears-level, on the margin for most people who read LessWrong.
Public mechanistic interpretability research is net positive in expectation.
Among existing alignment research agendas/projects, Superalignment has the highest expected value
Most LWers should rely less on norms of their own (or the LW community’s) design, and instead defer to regular societal norms more.
Rationalists would be better off if they were more spiritual/religious
Effective altruism can be well modeled by cynically thinking of it as just another social movement, in the sense that those a part of it are mainly jockeying for in-group status, and making costly demonstrations to their in-group & friends that they care about other sentiences more than others in the in-group. Its just that EA has more cerebral standards than others.
“Agent” is an incoherent concept.
“Open-source LLM-based agent with hacking abilities starts spreading itself over the Internet because some user asked it to do so or to do something like to conquer the world” is a quite probable point-of-no-return regarding AGI risk.
Investing in early-stage AGI companies helps with reducing x-risk (via mission hedging, having board seats, shareholder activism)
Great art is rarely original and mostly copied.
The younger generation of rationalists are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.
At least one of {Anthropic, OpenAI, Deepmind} is net-positive compared to the counterfactual where just before founding the company, its founders were all discretely paid $10B by a time-travelling PauseAI activist not to found the company and to exit the industry for 30 years, and this worked.
One should basically not invest into having “charisma”.
The most valuable new people joining AI safety will usually take ~1-3 years of effort to begin to be adequately sorted and acknowledged for their worth, unless they are unusually good at self-promotion e.g. gift of gab, networking experience, and stellar resume.
Poll feature on LW: Yay or Nay?
There is a greater than 80% chance that effective altruism has been net-negative for the world.
If rationality took off in China, it would yield higher EV from potentially spreading to the rest of the world than from potentially accelerating China.
When people try to discuss philosophy, math, or science, especially pre-paradigmatic fields such as ai safety, they use a lot of metaphorical thinking to extend from familiar concepts to new concepts. It would be very helpful and people would stop talking past each other so much if they practiced being explicitly aware of these mental representations and directly shared them rather than pretending that something more rigorous is happening. This is part of Alfred Korzybski’s original rationality project, something he called ‘consciousness of abstraction.’
Language model agents are very likely (>80%) to produce AGI (including the generalization to foundation model-based cognitive architectures)
Research into getting a mechanistic understanding of the brain for purposes of at least one of: understanding how values/empathy works in people, brain uploading or improving cryonics/plastination is net positive and currently greatly underfunded.
Most persistent disagreements can more usefully be thought of as a difference in priors rather than a difference in evidence or rationality.
A Secular Solstice variation designed to work weekly (akin to Sunday Service or Shabbat) would be positive for rationalists, both for community and for the thought processes of the members.
There is a greater than 50% chance that the Effective Altruism movement has been net negative for the world.
Most LWers are prioritizing their slack too much.
“Intelligence” can be characterized with a similar level of theoretical precision as e.g., heat, motion, and information. (In other words: it’s less like a messy, ad-hoc phenomena and more like a deep, general fact about our world).
You know of a technology that has at least a 10% chance of having a very big novel impact on the world (think the internet or ending malaria) that isn’t included in this list, very similar, or downstream from some element of it: AI, mind uploads, cryonics, human space travel, geo-engineering, gene drives, human intelligence augmentation, anti-aging, cancer cures, regenerative medicine, human genetic engineering, artificial pandemics, nuclear weapons, proper nanotech, very good lie detectors, prediction markets, other mind-altering drugs, cryptocurrency, better batteries, BCIs, nuclear fusion, better nuclear fission, better robots, AR, VR, room-temperature superconductors, quantum computers, polynomial time SAT solvers, cultured meat, solutions to antibiotic resistance, vaccines to some disease, optical computers, artificial wombs, de-extinction and graphene.
Bad options included just in case someone thinks they are good.
Xi Jinping thinks that economic failure in the US or China, e.g. similar to 2008, is one of the most likely things to change the global balance of power.
Having another $1 billion to prevent AGI x-risk would be pretty useful.
If we had access to a brain upload (and maybe a world simulator too) we could in principle extract something like a utility function, and the theory behind it relates more to agents in general than it does to humans in particular.
Any activity or action taken after drinking coffee in the morning will strongly reward/reinforce that action/activity
Humans are the dominant species on earth primarily because our individual intelligence surpassed the necessary threshold to sustain civilization and take control of our environment.
American intelligence agencies are actively planning to defend the American AI industry against foreign threats (e.g. Russia, China).
If you can write prompt for GPT-2000 such that completion of this prompt results in aligned pivotal act, you can just use knowledge necessary for writing this prompt to Just Build aligned ASI, without necessity to use GPT-2000.
Rationalist rituals like Petrov day or the Secular Solstices should be marginally more emphasized within those collections of people who call themselves rationalists.
Rationality is likely to organically gain popularity in China (e.g. quickly reaching 10,000 people or reaching 100,000 by 2030, e.g. among scientists or engineers, etc).
It is wrong to protest AI labs.
The ratio of good alignment work done at labs vs in academia mostly skews toward labs
Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than academics. Your answer should factor out such considerations.
Edit: Also normalize for quantity of researchers.
The ratio of good alignment work done in academia vs independently mostly skews toward academia
Good meaning something different from impactful here. Possibly AGI labs will pay more attention to academics than independent researchers. Your answer should factor out such considerations.
Edit: Also normalize for quantity of researchers.
At least one mole, informant, or spy has been sent by a US government agency or natsec firm to infiltrate the AI safety community by posing as a new member (even if it’s just to ask questions in causal conversations at events about recent happenings or influential people’s priorities).
The government should build nuclear-driven helicopters, like nuclear subs.
An alignment technique that can fully align GPT-4 is likely (>50%) to also fully align the first existentially dangerous AGI
Most end-to-end “alignment plans” are bad because research will be incremental. For example, Superalignment’s impact will mostly come from adapting to the next ~3 years of AI discoveries and working on relevant subproblems like interp, rather than creating a superhuman alignment researcher.
Conceptual alignment work on concepts like “agency”, “optimization”, “terminal values”, “abstractions”, “boundaries” is mostly intractable at the moment.
Most LWers are not prioritizing their slack enough.
Those who call themselves rationalists or EAs should drink marginally more alcohol at social events.
In human interactions at any scale, it is net good to at least momentarily consider the Elephant and/or the Player, with very few exceptions.
Most of the time, power-seeking behavior in humans is morally good or morally neutral.
Computer science & ML will become lower in relevance/restricted in scope for the purposes of working with silicon-based minds, just as human-neurosurgery specifics are largely but not entirely irrelevant for most civilization-scale questions like economic policy, international relations, foundational research, etc.
Or IOW: Model neuroscience (and to some extent, model psychology) requires more in-depth CS/ML expertise than will the smorgasbord of incoming subfields of model sociology, model macroeconomics, model corporate law, etc.
The virtue of the void is indeed the virtue above all others (in rationality), and fundamentally unformalizable.
There is likely a deep compositional structure to be found for alignment, possibly to the extent that AGI alignment could come from “merely” stacking together “microalignment”, even if in non-trivial ways.
If you can’t write a program that produces aligned (under whatever definition of alignment you use) output being run on unphysically large computer, you can’t deduce from training data or weights of superintelligent neural network if it produces aligned output.
There are arguments for convergent instrumental pressures towards catastrophe, but the required assumptions are too strong for the arguments to clearly go through.
Cultural values are something like preferences over pairs of social environments and things we actually care about. So it makes sense to talk about jointly optimizing them.
It’s good for EA orgs to pay well
The younger generation of EAs are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.
In the context of Offense-defense balance, offense has a strong advantage
Generative AI like LLMs or diffusion will eventually be superseded by human AI researchers coming up with something autonomous.
EA has gotten a little more sympathetic to vibes-based reasoning recently, and will continue to incorporate more of it.
The mind (ie. your mind), and how it is experienced from the inside, is potentially a very rich source of insights for keeping AI minds aligned on the inside.
All else equal, a unit of animal suffering should be accorded the same moral weight as an equivalent unit of human suffering. (i.e. equal consideration for equal interests)
‘Descriptive’ agent foundations research is currently more important to work on than ‘normative’ agent foundations research.
Autism is the extreme male version of a male-female difference in systemic vs empathic thinking.
Developing a solid human intelligence/skill evaluation metric would be a high-EV project for AI safety, e.g. to make it easier to invest in moving valuable AI safety people to the Bay Area/London from other parts of the US/UK.
MadHatter is an original and funny satirist. The universally serious reaction to his jokeposts is a quintessential example of rationalist humorlessness.
People should pay an attractiveness tax to the government.