Co-Founder & Director—AI Safety ANZ (join us: www.aisafetyanz.com.au)
Advisory Board Member (Growth) - Giving What We Can
Creating Superintelligent Artificial Agents (SAA) without a worldwide referendum is ethically unjustifiable. Until a consensus is reached on whether to bring into existence such technology, a global moratorium is required (we already have AGI).
AI Safety has less money, talent, political capital, tech and time. We have only one distinct advantage: support from the general public. We need to start working that advantage immediately.
Solving the AGI alignment problem demands a herculean level of ambition, far beyond what we’re currently bringing to bear. Dear Reader, grab a pen or open a google doc right now and answer these questions:
1. What would you do right now if you became 5x more ambitious?
2. If you believe we all might die soon, why aren’t you doing the ambitious thing?
One idea: Create a LinkedIn advertiser account and segment by Industry and or Job Title. and or Job Function
I think > 40% of AI Safety resources should be going into making Federal Governments take seriously the possibility of an intelligence explosion in the next 3 years due to proliferation of digital agents.
SASH isn’t official (we’re waiting on funding).
Here is TARA :)
https://www.lesswrong.com/posts/tyGxgvvBbrvcrHPJH/apply-to-be-a-ta-for-tara
I think it will take less than 3 years for the equivalent of 1,000,000 people to get laid off.
If transformative AI is defined by its societal impact rather than its technical capabilities (i.e. TAI as process not a technology), we already have what is needed. The real question isn’t about waiting for GPT-X or capability Y—it’s about imagining what happens when current AI is deployed 1000x more widely in just a few years. This presents EXTREMELY different problems to solve from a governance and advocacy perspective.
E.g. 1: compute governance might no longer be a good intervention
E.g. 2: “Pause” can’t just be about pausing model development. It should also be about pausing implementation across use cases
Yep
E.g. 50,000 travel agents lose their jobs in 25⁄26
That’s pretty high. What use cases are you imagining as the most likely?
I’m not sure of an org that deals with ultra-high net worth individuals (longview?), but someone should reach out to Bryan Johnson. I think he could be persuaded to invest in AI Safety (skip to 1:07:15)
I spent 8 years working in strategy departments for Ad Agencies. If you’re interested in the science behind brand tracking, I recommend you check out the Ehrenberg-Bass Institutes work on Category Entry Points: https://marketingscience.info/research-services/identifying-and-prioritising-category-entry-points/
What is your AI Capabilities Red Line Personal Statement? It should read something like “when AI can do X in Y way, then I think we should be extremely worried / advocate for a Pause*”.
I think it would be valuable if people started doing this; we can’t feel when they’re on an exponential, so its likely we will have powerful AI creep up on us.
@Greg_Colbourn just posted this and I have an intuition that people are going to read it and say “while it can do Y it still can’t do X”
*in the case you think a Pause is ever optimal.
I’d like to see research that uses Jonathan Haidt’s Moral Foundations research and the AI risk repository to forecast whether there will soon be a moral backlash / panic against frontier AI models.
Ideas below from Claude:
# Conservative Moral Concerns About Frontier AI Through Haidt’s Moral Foundations
## Authority/Respect Foundation
### Immediate Concerns
- AI systems challenging traditional hierarchies of expertise and authority
- Undermining of traditional gatekeepers in media, education, and professional fields
- Potential for AI to make decisions traditionally reserved for human authority figures
### Examples
- AI writing systems replacing human editors and teachers
- AI medical diagnosis systems challenging doctor authority
- Language models providing alternative interpretations of religious texts
## Sanctity/Purity Foundation
### Immediate Concerns
- AI “thinking” about sacred topics or generating religious content
- Artificial beings engaging with spirituality and consciousness
- Degradation of “natural” human processes and relationships
### Examples
- AI spiritual advisors or prayer companions
- Synthetic media creating “fake” human experiences
- AI-generated art mixing sacred and profane elements
## Loyalty/Ingroup Foundation
### Immediate Concerns
- AI systems potentially eroding national sovereignty
- Weakening of traditional community bonds
- Displacement of local customs and practices
### Examples
- Global AI systems superseding national control
- AI replacing local business relationships
- Automated systems reducing human-to-human community interactions
## Fairness/Reciprocity Foundation
### Immediate Concerns
- Unequal access to AI capabilities creating new social divides
- AI systems making decisions without clear accountability
- Displacement of workers without reciprocal benefits
### Examples
- Elite access to powerful AI tools
- Automated systems making hiring/firing decisions
- AI wealth concentration without clear social benefit
## Harm/Care Foundation
### Immediate Concerns
- Potential for AI systems to cause unintended societal damage
- Psychological impact on children and vulnerable populations
- Loss of human agency and autonomy
### Examples
- AI manipulation of emotions and behavior
- Impact on childhood development
- Dependency on AI systems for critical decisions
## Key Trigger Points for Conservative Moral Panic
1. Religious Content Generation
- AI systems creating or interpreting religious texts
- Automated spiritual guidance
- Virtual religious experiences
2. Traditional Value Systems
- AI challenging traditional moral authorities
- Automated ethical decision-making
- Replacement of human judgment in moral domains
3. Community and Family
- AI impact on child-rearing and education
- Disruption of traditional social structures
- Replacement of human relationships
4. National Identity
- Foreign AI influence on domestic affairs
- Cultural homogenization through AI
- Loss of local control over technology
## Potential Flashpoints for Immediate Backlash
1. Education System Integration
- AI teachers and tutors
- Automated grading and assessment
- Curriculum generation
2. Religious Institution Interaction
- AI pastoral care
- Religious text interpretation
- Spiritual counseling
3. Media and Information Control
- AI content moderation
- Synthetic media generation
- News creation and curation
4. Family and Child Protection
- AI childcare applications
- Educational AI exposure
- Family privacy concerns
I think there is a 10-20 per cent chance we get digital agents in 2025 that produce a holy shit moment as big as the launch of chatgpt.
If that happens I think that will produce another round of questions that sounds approximately like “how were we so unprepared for this moment”.
Fool me once, shame on you…
I am 90% sure that most AI Safety talent aren’t thinking hard enough about what Neglectedness. The industry is so nascent that you could look at 10 analogous industries, see what processes or institutions are valuable and missing and build an organisation around the highest impact one.
The highest impact job ≠ the highest impact opportunity for you!
AI Safety (in the broadest possible sense, i.e. including ethics & bias) is going be taken very seriously soon by Government decision makers in many countries. But without high quality talent staying in their home countries (i.e. not moving to UK or US), there is a reasonable chance that x/c-risk won’t be considered problems worth trying to solve. X/c-risk sympathisers need seats at the table. IMO local AIS movement builders should be thinking hard about how to keep talent local (this is an extremely hard problem to solve).
One axis where Capabilities and Safety people pull apart the most, with high consequences is on “asking for forgiveness instead of permission.”
1) Safety people need to get out there and start making stuff without their high prestige ally nodding first
2) Capabilities people need to consider more seriously that they’re building something many people simply do not want