Interesting. I wasn’t aware of the specific equations, but at a first glance, that does seem like a reasonable argument—thanks!
Also, to follow up on my second comment (sorry, the formatting was a bit confusing but I just edited it): I think it would’ve been valuable for the blog post to end with a more concrete “bridge” to the solutions as the current version is fairly generic.
As mentioned in a previous comment, it seems that many of the concepts and claims are fairly well-established in these fora already (though I appreciated the concrete examples of the Darwinian demons and Darwinians angels throughout history and some of the exhibits provided in favor of the Fragility of Life Hypotheses).
What do you think are the best arguments against the Fragility of Life Hypotheses?
we have the chance to shape our future with rationality, empathy, and foresight—a potential savior not just for ourselves, but for all life.
This struck me as a bit vague, but I’m guessing that you’ll be much more specific about potential solutions in your upcoming blog post.
Could you share more about this:
“my own experiences suggest that many policymakers appear to be very interested in better understanding loss of control scenarios.”
For example, what proportion of people you’ve interacted with gave you this impression and the context in which it happened (I understand that you’d have to keep details sparse for obvious reasons).
Congrats on publishing the book. It seems that many of the concepts you discuss seem well-known (especially within EA and LW). However, am I right to say that the novelty/contribution of the work lies in the following (caveat—obviously, this is a shallow summary):
1. Making multipolar traps, stemming from the evolutionary optimization process, the central concept (instead of doing the most good or becoming less wrong).
2. Outlining how it’s the root cause of key priorities such as existential risks (and perhaps things such as the Fermi paradox).
3. Describing how multipolar traps can be avoided via global coordination in the form of value chains and related mechanisms. If so, this might be the most novel part of the book.
I think I disagree with some of the claims in this post and I’m mostly sympathetic with the points Akash raised in his comments. Relatedly, I’d like to see a more rigorous comparison between the AI safety community (especially EA/Rationality parts) and relevant reference class movements such as the climate change community.
That said, I think it’s reasonable to have a high prior on people ending up aiming for inappropriate levels of power-seeking when taking ambitious actions in the world so it’s important to keep these things in mind.
In addition to your two “recommendations” of focusing on legitimacy and competence, I’d add two additional candidates:
1. Being careful about what de facto role models or spokespeople the AI safety community “selects”. It seems crucial to avoid another SBF.
2. Enabling currently underrepresented perspectives to contribute in well-informed, competent ways.
I agree that international coordination seems very important.
I’m currently unsure about how to best think about this. One way is to focus on bilateral coordination between the US and China, as it seems that they’re the only actors who can realistically build AGI in the coming decade(s).
Another way, is to attempt to do something more inclusive by also focusing on actors such as UK, EU, India, etc.
An ambitious proposal is the Multinational AGI Consortium (MAGIC). It clearly misses many important components and considerations, but I appreciate the intention and underlying ambition.
Thanks for this—I wasn’t aware. This also makes me more disappointed with the voluntary commitments at the recent AI Safety Summit. As far as I can tell (based on your blog post), the language used around external evaluations was also quite soft:
”They should also consider results from internal and external evaluations as appropriate, such as by independent third-party evaluators, their home governments[3], and other bodies their governments deem appropriate.”
I wonder if it’d be possible to have much stronger language around this at next year’s Summit.
The EU AI Act, is the only regulatory rules that isn’t voluntary (i.e., companies have to follow them) and even that doesn’t require external evaluations (they use the wording Conformity Assessment) for all high-risk systems.
Any takes on what our best bet is for making high-quality external evals mandatory for frontier models at this stage?
This is a valid concern, but I’m fairly certain Science (the journal—not the field) handled this well. Largely because they’re somewhat incentivized to do so (otherwise it could have very bad optics for them) and must have experienced this several times before. I also happen to know one of the senior authors who is significantly above average in conscientiousness.
Good point. Just checked—here’s the ranking of Chinese LLMs among the top 20:
7. Yi-Large (01 AI)
12. Qwen-Max-0428 (Alibaba)
15. GLM-4-0116 (Zhipu AI)
16. Qwen1.5-110B-Chat (Alibaba)
19. Qwen1.5-72B-chat (Alibaba)
Of the three, only Zhipu signed the agreement.
Thanks for this analysis—I found it quite helpful and mostly agree with your conclusions.
Looking at the list of companies who signed the commitments, I wonder why Baidu isn’t on there. Seems very important to have China be part of the discussion, which, AFAIK, mainly means Baidu.
I am honored to be part of enabling more people from around the world to contribute to the safe and responsible development of AI.
Thanks for posting.
The hindrances you mention seem relevant but also highly related. In particular, the
”Poorly-tested implementation: A disconnection between engaging educational content and its practical application, underscoring the importance of field-tested guidance that works in varied real-life situations.”
I’m excited to see advice that has been tested via working closely with individuals. I’m reminded of the Scott Miller (who played a major role within evidence-based psychotherapy) claim on a podcast (paraphrasing): “We have an implementation problem”.
Great! I’d expect most people on there are. I know for sure that Paul Rohde and James Norris (the founder) are aware. My rates depends on the people I work with but $200-$300 is the standard rate.
Thank you. This is a really excellent post. I’d like to add a few resources and providers:
1. EA mental health navigator: https://www.mentalhealthnavigator.co.uk/.
2. Overview of providers on EA mental health navigator (not everyone familiar with alignment in significant ways). https://www.mentalhealthnavigator.co.uk/providers
3. Upgradable has some providers that are quite informed around alignment. https://www.upgradable.org/
4. If permissible, I’d like to add myself as a provider (coach) though I don’t take on any coachees at present.
So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies ‘belief,’ and the latter thingy ’reality.
Re-reading this after four years. It is still so elegantly beautiful. I don’t understand why this wasn’t part of sequences highlights. I would have put under “thinking better on purpose” or “the laws governing beliefs”.
Interesting that you seem to see rationality (as opposed to traditional rationality) as a more effective and efficient version of seeking the truth (~epistemic rationality). In that sense, it does seem somewhat similar to doing what EA is trying to do for altruism.
Big +1 on adding this and/or finding another high-quality way of depicting what the ideal scenario would look like. I think many people think and feel that the world is in a very dire state to an extent that leads to hopelessness and fatalism. Articulating clear theories of victory that enable people to see the better future they can contribute towards will be an important part of avoiding this scenario.