Request: Put Carl Shulman’s recent podcast into an organized written format

[Aryeh Englander]

Carl Shulman recently did a very long interview with Dwarkesh Patel on the Lunar Society Podcast, which Dwarkesh released in two separate episodes because of its length. I have seen many people point to this interview as adding a lot of important high value content to the AI x-risk discussion, and I agree with that assessment. However, I don’t think it’s fair to tell people they need to listen to a 7 hour interview in order to get a better sense of concrete risk scenarios.

I therefore think it would be very high value if someone could organize the information that Carl presented into an accessible written format so that it can be used as a reference going forward. I think it would be even better if Carl could publish it on arxiv or in some relevant journal so that it could get the academic attention it deserves.

[Raemon]

As an incremental step here, I made two google docs for the Pt 1 and Pt 2 transcript, where people can comment on bits that seemed particularly interesting. This seemed like a helpful step for thinking through the transcript.

Part 1

Part 2

[Sameerishere]

Dwarkesh Patel seems to have a full transcript on his blog https://​​www.dwarkeshpatel.com/​​p/​​carl-shulman#details

[Moonicker]

I have done so. However as mentioned in the comments Dwarkesh has done so prior. This is basically just a copy and paste with some minor editing and packaging.

https://​​www.lesswrong.com/​​posts/​​BdPjLDG3PBjZLd5QY/​​carl-shulman-on-dwarkesh-podcast-june-2023

[bvbvbvbvbvbvbvbvbvbvbv]

edit: I can make the prompt more or less compressed easily, just ask. The present example is “pretty compressed” but I can make a more verbose one

Not really what you’re asking but :

I’m coincidentally working on the side on a DIY summarizer to manage my inputs. I summarized a bit of the beginning of part 1. If you think it has any value I can run the whole thing :

note that ‘- ---’ indicate the switch to a new chunk of text by the llm

This is formatted as a logseq /​ obsidian markdown format.

- Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment - https://​​youtube.com/​​watch?v=_kRg-ZP1vQc
  summarization_date:: 28/​06/​2023
  token_cost:: 12057
  dollar_cost:: 0.01911
  summary_reading_length:: 4.505
  doc_reading_length:: 120.5025
  author:: Dwarkesh Patel
    - Carl Shulman: highly regarded intellectual known for ideas on intelligence explosion and its impacts
      - Advisor to Open Philanthropy project
      - Research associate at Future of Humanity Institute at Oxford
    - Feedback loops and dynamics when approaching human-level intelligence involve:
      - Development of new computer chips, software, and training runs
    - Concept of input-output curves important in understanding increasing difficulty of improving AI
    - Productivity of computing has increased significantly over the years, but investment and labor required for advancements have also increased
    - In a world where AI is doing the work, doubling computing performance translates to a doubling or better of effective labor supply
    - Doubling labor force can result in several doublings of compute, accelerating AI development
    - Bloom paper mentioned:
      − 35% increase in transistor density
      − 7% increase per year in number of researchers required to sustain that pace
    - ---
    - The bloom paper mentioned:
      − 35% increase in transistor density
      − 7% increase per year in the number of researchers required to sustain that pace
    - There is a question of whether AI can be seen as a population of researchers that grows with compute itself.
    - Compute is a good proxy for the number of AI researchers because:
      - If you have an AI worker that can substitute for a human, having twice as many computers allows for running two separate instances and getting more gains.
    - Improvements in hardware and software technology contribute to the progress of AI.
    - The work involved in designing new hardware and software is done by people, but computer time is not the primary cost.
    - The number of people working on AI research is in the low tens of thousands, with companies like Nvidia, TSMC, and DeepMind having significant numbers of employees.
    - The capabilities of AI are doubling on a shorter time scale than the number of people required to develop them.
    - ---
    - The capabilities of AI are doubling faster than the number of people needed to develop them.
      - Hardware efficiency has historically doubled 4-5 times per doubling of human inputs, but this rate has slowed down as Moore’s Law nears its end.
      - On the software side, the doubling time for workers driving software advances is several years, while the doubling time for effective compute from algorithmic progress is faster.
    - Epoch, a group that collects datasets relevant to forecasting AI progress, found the following doubling times:
      - Hardware efficiency doubles in about 2 years.
      - Budget growth doubles in about 6 months.
      - Algorithmic progress doubles in less than 1 year.
    - The growth of effective compute for training big AIs is drastic, with estimates that GPT-4 cost around 50 million dollars to train.
      - Effective compute can increase through greater investment, better models, or cheaper training chips.
    - Software progress is measured by the reduction in compute needed to achieve the same benchmark as before.
    - The feedback loop between AI and compute can help with hardware design and chip improvements.
    - Automating chip design work could lead to faster improvements, but it is less important for the intelligence explosion.
    - ---
    - Improving chip design through AI automation is less important for the intelligence explosion because it only applies to future chips.
      - Faster improvements can be achieved through AI automation.
    - The most disruptive and important aspect of AI automation is on the software side.
      - Improvements can be immediately applied to existing GPUs.
    - The question is when AI will contribute significantly to AI progress and software development.
      - This contribution could be equivalent to having additional researchers.
    - The magnitude of AI’s contribution is crucial.
      - It should boost effective productivity by 50-100% or more.
    - AI can automate certain tasks in the AI research process.
      - This allows for more frequent and cost-effective completion of these tasks.
    - The goal is to have AI that can significantly enhance performance.
      - This is even with its weaknesses, rather than achieving human-level AI with no weaknesses.
    - Existing fabs can produce tens of millions of advanced GPUs per year.
      - If they run AI software as efficient as humans, with extended work hours and education, it can greatly surpass human capabilities.
    - ---
    - The education level of AI models surpasses that of humans and focuses on specific tasks.
      - Tens of millions of GPUs, each equivalent to the work of the best humans, contribute to significant discoveries and technological advancements.
      - Human-level AI is currently experiencing an intelligence explosion, starting from a weaker state.
      - The feedback loop for AI researchers begins when they surpass small productivity increases and reach a level equivalent to or close to human researchers.
      - AI systems can compensate for their weaknesses by deploying multiple less intelligent AIs to match the capabilities of a human worker.
      - AI can be applied to tasks such as voting algorithms, deep search, and designing synthetic training data, which would be impractical for humans.
      - As AI becomes more advanced, it can generate its own data and identify valuable skills to practice.
      - For instance, AlphaZero generated its own data through self-play and followed a curriculum to always compete against an opponent of equal skill.