In defence of attempting unnatural or extreme strategies
As a response to Rob’s post, this sounds too me like it’s misunderstanding what he was pointing out. I don’t think he was pointing to weird ideas — I’m pretty sure he knows very well we need new ideas to solve alignment. What he was pointing out was people who are so panicked that they start discussing literal crimes or massive governance interventions that are incredibly hard to predict.
Upvoted because I think there should be more of a discussion around this then “Obviously getting normal people involved will only make things worse” (which seems kind of arrogant / assumes there are no good unknown unknowns)
Yes, I’m not convinced either way myself but here are some arguments against:
If the USA regulates AGI, China will get it first which seems worse as there’s less alignment-activity in China (as for US China coordination, lol, lmao)
Raising awareness of AGI Alignment also raises awareness of AGI. If we communicate the “AGI” part without the “Alignment” part we could speed up timelines
If there’s a massive influx of funding/interest from people who aren’t well informed, it could lead to “substitution hazards” like work on aligning weak models with methods that don’t scale to the superintelligent case (In climate change people substitute “solve climate change” to “I’ll reduce my own emissions” which is useless)
If we convince the public AGI is a threat, there could be widespread flailing (the bad kind) which reflects badly on Alignment researchers (e.g. if DeepMind researchers are receiving threats, their system 1 might generalize to “People worried about AGI are a doomsday cult and should be disregarded”)
Most of these I’ve heard from reading conversations on EleutherAI’s discord, Connor is typically the most pessimistic but some others are pessimistic too (Connor’s talk discusses substitution hazards in more detail)
TLDR: It’s hard to control the public once they’re involved. Climate change startups aren’t getting public funding, the public is more interested in virtue-signaling (In the climate case the public doesn’t really make things worse, but for AGI it could be different)
EDIT: I think I’ve presented the arguments badly, re-reading them I don’t find them convincing. You should seek out someone who presents them better.
If you’re in the Markdown editor, you can make footnotes instead of *, **, ***. Do that by making one of these → [^1] (ex output: [1]) and then at the bottom, put [^1]: Your footnote here
It doesn’t have to be just numbers, it can also be a word like [^example] or [^defense_of_virtue]. The words are nice because they look good in a link.
If you genuinely believe that the world is ending in 20 years, but are not visibily affected by this, or considering extreme actions, people may be less likely to believe that you believe what you say you do.
IMO, that’s not the bottleneck. The bottleneck is people thinking you’re insane, which composure mitigates.
It feels more to me like we’re the quiet weird kid in high school that doesn’t speak up or show emotion because we’re afraid of getting judged or bullied. Which, fair enough, the school is sort of like—just look at poor cryonics, or even nuclear power—but the road to popularity (let along getting help with what’s bugging us) isn’t to try to minimize our expressions to ‘proper’ behavior while letting us be characterized by embarrassing past incidents (e.g. Roko’s Basilisk) if we’re noticed at all.
It isn’t easy to build social status, but right now we’re trying next to nothing and we’ve seen it doesn’t seem to do enough.
As a response to Rob’s post, this sounds too me like it’s misunderstanding what he was pointing out. I don’t think he was pointing to weird ideas — I’m pretty sure he knows very well we need new ideas to solve alignment. What he was pointing out was people who are so panicked that they start discussing literal crimes or massive governance interventions that are incredibly hard to predict.
Upvoted because I think there should be more of a discussion around this then “Obviously getting normal people involved will only make things worse” (which seems kind of arrogant / assumes there are no good unknown unknowns)
Yes, I’m not convinced either way myself but here are some arguments against:
If the USA regulates AGI, China will get it first which seems worse as there’s less alignment-activity in China (as for US China coordination, lol, lmao)
Raising awareness of AGI Alignment also raises awareness of AGI. If we communicate the “AGI” part without the “Alignment” part we could speed up timelines
If there’s a massive influx of funding/interest from people who aren’t well informed, it could lead to “substitution hazards” like work on aligning weak models with methods that don’t scale to the superintelligent case (In climate change people substitute “solve climate change” to “I’ll reduce my own emissions” which is useless)
If we convince the public AGI is a threat, there could be widespread flailing (the bad kind) which reflects badly on Alignment researchers (e.g. if DeepMind researchers are receiving threats, their system 1 might generalize to “People worried about AGI are a doomsday cult and should be disregarded”)
Most of these I’ve heard from reading conversations on EleutherAI’s discord, Connor is typically the most pessimistic but some others are pessimistic too (Connor’s talk discusses substitution hazards in more detail)
TLDR: It’s hard to control the public once they’re involved. Climate change startups aren’t getting public funding, the public is more interested in virtue-signaling (In the climate case the public doesn’t really make things worse, but for AGI it could be different)
EDIT: I think I’ve presented the arguments badly, re-reading them I don’t find them convincing. You should seek out someone who presents them better.
I suspect that mass outreach is likely to be too slow to make a big difference and so may not be worth it given the possible downsides.
That said, I am in favour of addressing public misconceptions rather than just letting them stand.
Note about formatting:
If you’re in the Markdown editor, you can make footnotes instead of *, **, ***. Do that by making one of these →
[^1](ex output: [1]) and then at the bottom, put[^1]: Your footnote hereIt doesn’t have to be just numbers, it can also be a word like
[^example]or[^defense_of_virtue]. The words are nice because they look good in a link.IMO, that’s not the bottleneck. The bottleneck is people thinking you’re insane, which composure mitigates.
It feels more to me like we’re the quiet weird kid in high school that doesn’t speak up or show emotion because we’re afraid of getting judged or bullied. Which, fair enough, the school is sort of like—just look at poor cryonics, or even nuclear power—but the road to popularity (let along getting help with what’s bugging us) isn’t to try to minimize our expressions to ‘proper’ behavior while letting us be characterized by embarrassing past incidents (e.g. Roko’s Basilisk) if we’re noticed at all.
It isn’t easy to build social status, but right now we’re trying next to nothing and we’ve seen it doesn’t seem to do enough.
Both, I’d think.