I am strong down voting in this case as when I put a noticeable amount of effort responding to your prior post “are there 2 types of alignment?”, you gave an unsubstantiative followup to my answer to your question, and no followup to the 5 other people who commented in response to your post.
When I attempted to communicate with you clearly and helpfully in response to one of your low effort questions, I saw little value. Why should others listen to you when you tell them to do what I did?
Robert Cousineau
Karma: 158
I quite enjoyed reading this—I’m surprised I’d not read something like it before and quite happy you did the work and posted it here.
Do you have plans of using the dataset you built here to work on “figuring out if AI is conscious”?
Agreed—that’s what I was trying to say with the link under “80b number is the same number Microsoft has been saying repeatedly.”
That would be described well by the CEV link above.
I think having a single word like “Alignment” mean multiple things is quite useful, similar to how I think having a single word like “Dog” mean many things is also useful.
I’m having trouble remembering many times people here say “AI Alignment” in a way that would be best described as “making an AI that builds utopia and stuff”. Maybe Coherent Extrapolated Volition would be close.
My general understanding is that when people here talk about AI Alignment, they are talking about something closer to what you call “making an AI that does what we mean when we say ‘minimize rate of cancer’ (that is, actually curing cancer in a reasonable and non-solar-system-disassembling way)”.
On a somewhat related point, I’d say that “making an AI that does what we mean when we say “minimize rate of cancer” (that is, actually curing cancer in a reasonable and non-solar-system-disassembling way)” is entirely encapsulated under “making an AI that builds utopia and stuff”, as it is very very unlikely an AI makes a utopia while misunderstanding what we intended its goal to be that much.
You would likely enjoy reading through this (short) post: Clarifying inner alignment terminology, and I expect it would help you get a better understanding of what people mean when they are discussing AI Alignment.
Another resource you might enjoy would be reading through the Tag and Subtags around AI: https://www.lesswrong.com/tag/ai
PS: In the future, I’d probably make posts like this in the Open Thread.
Here is what I posted on “Quotes from the Stargate Press Conference”:
On Stargate as a whole:
This is a restatement with a somewhat different org structure of the prior OpenAI/Microsoft data center investment/partnership, announced early last year (admittedly for $100b).
Elon Musk states they do not have anywhere near the 500 billion pledged actually secured:I do take this as somewhat reasonable, given the partners involved just barely have $125 billion available to invest like this on a short timeline.
Microsoft has around 78 billion cash on hand at a market cap of around 3.2 trillion.
Softbank has 32 billion dollars cash on hand, with a total market cap of 87 billion.
Oracle has around 12 billion cash on hand, with a market cap of around 500 billion.
OpenAI has raised a total of 18 billion, at a valuation of 160 billion.
Further, OpenAI and Microsoft seem to be distancing themselves somewhat—initially this was just an OpenAI/Microsoft project, and now it involves two others and Microsoft just put out a release saying “This new agreement also includes changes to the exclusivity on new capacity, moving to a model where Microsoft has a right of first refusal (ROFR).”
Overall, I think that the new Stargate numbers published may (call it 40%) be true, but I also think there is a decent chance this is new administration trump-esque propoganda/bluster (call it 45%), and little change from the prior expected path of datacenter investment (which I do believe is unintentional AINotKillEveryone-ism in the near future).
Edit: Satya Nadella was just asked about how funding looks for stargate, and said “Microsoft is good for investing 80b”. This 80b number is the same number Microsoft has been saying repeatedly.
On Stargate as a whole:
This is a restatement with a somewhat different org structure of the prior OpenAI/Microsoft data center investment/partnership, announced early last year (admittedly for $100b).
Elon Musk states they do not have anywhere near the 500 billion pledged actually secured:I do take this as somewhat reasonable, given the partners involved just barely have $125 billion available to invest like this on a short timeline.
Microsoft has around 78 billion cash on hand at a market cap of around 3.2 trillion.
Softbank has 32 billion dollars cash on hand, with a total market cap of 87 billion.
Oracle has around 12 billion cash on hand, with a market cap of around 500 billion.
OpenAI has raised a total of 18 billion, at a valuation of 160 billion.
Further, OpenAI and Microsoft seem to be distancing themselves somewhat—initially this was just an OpenAI/Microsoft project, and now it involves two others and Microsoft just put out a release saying “This new agreement also includes changes to the exclusivity on new capacity, moving to a model where Microsoft has a right of first refusal (ROFR).”
Overall, I think that the new Stargate numbers published may (call it 40%) be true, but I also think there is a decent chance this is new administration trump-esque propoganda/bluster (call it 45%), and little change from the prior expected path of datacenter investment (which I do believe is unintentional AINotKillEveryone-ism in the near future).
Edit: Satya Nadella was just asked about how funding looks for stargate, and said “Microsoft is good for investing 80b”. This 80b number is the same number Microsoft has been saying repeatedly.
As best I can tell, the US AI Safety institute is likely to be shuttered in the near future. I bet accordingly on this market.
Trump rescinded Executive Order 14110, which established the U.S. AI Safety Institute (AISI).
There was some congressional work going on (HR 9497 and S.4769) and that would have formalized the AISI outside of the executive order, but that has not been enacted per my best understanding of the machinations of our government.
Here’s to hoping I’m wrong! also maybe next time I’ll place a smaller bet...Edit: I sold my shares at a modest profit. It seems the AISI is less directly linked to 14110 than I expected. Further, no news on it yet seems unlikely if it was actually ending.
Robert Cousineau’s Shortform
I personally put a relatively high probability of this being a galaxy brained media psyop by OpenAI/Sam Altman.
Eliezer makes a very good point that confusion around people claiming AI advances/whistleblowing benefits OpenAI significantly, and Sam Altman has a history of making galaxy brained political plays (attempting to get Helen fired (and then winning), testifying to congress that it is good he has oversight via the board and he should not be full control of OpenAI and then replacing the board with underlings, etc).
Sam is very smart and politically capable. This feels in character.
I often find the insinuations people make with this graph to be misleading. The increase in time spent at home is in very large part due to the rise in remote work, which I would say is a public good (and at least for me, leads to much easier high quality socialization, as I can make my schedule work for my friends). Additionally. time spent not at home includes people commuting, with all of the negative internalities (risk of crash, wasted time, etc) and negative externalities (emissions, greater traffic load, etc) that driving includes.
That it is trending down post covid seems like a negative, not a positive.
I mostly agree with the body of this post, and think your calls to action make sense.
On your title and final note: Butlerian Jihad feels out of place. It’s catchy, but it seems like you are recommending AI concerned people more or less do what AI concerned people already do. I feel like we should save our ability to use words that are a call to arms for a time when that is what we are doing.
I’ve donated 5k. Lesswrong (and the people it brings together) deserve credit for the majority of my intellectual growth over the last 6 years. I cannot think of a higher signal:noise place to learn, nor can I think of a more enjoyable and growth inducing community than the community which has grown around it.
Thank you to both those who directly work on it and those who contribute to it!
Lighthaven’s wonder is self evident.
Wagering on Will And Worth (Pascals Wager for Free Will and Value)
I’m honestly really skeptical of the cost effectiveness of pedestrian tunnels as a form of transportation. Asking Claude for estimates on tunnel construction costs gets me the following:
A 1-mile pedestrian tunnel would likely cost $15M-$30M for basic construction ($3,000-$6,000 per foot based on utility tunnel costs), plus 30% for ventilation, lighting, and safety systems ($4.5M-$9M), and ongoing maintenance of ~$500K/year.To put this in perspective: Converting Portland’s 400 miles of bike lanes to tunnels would cost $7.8B-$15.6B upfront (1.1-2.3× Portland’s entire annual budget) plus $200M/year in maintenance. For that same $15.6B, you could:
Build ~780 miles of protected surface bike lanes ($2M/mile)
Fund Portland’s bike infrastructure maintenance for 31 years
Give every Portland resident an e-bike and still have $14B left over
Even for a modest 5-mile grid serving 10,000 daily users (optimistic for suburbs), that’s $10K-$20K per user in construction costs alone.
Alternative: A comprehensive street-level mural program might cost $100K-$200K per mile, achieving similar visual variety at ~1% of the tunnel cost.
I’ll preface this with: what I’m saying is low confidence—I’m not very educated on the topics in question (reality fluid, consciousness, quantum mechanics, etc).
Nevertheless, I don’t see how the prison example is applicable. In the prison scenario there’s an external truth (which prisoner was picked) that exists independent of memory/consciousness. The memory wipe just makes the prisoner uncertain about this external truth.
But this post is talking about a scenario where your memories/consciousness are the only thing that determines which universes count as ‘you’.
There is no external truth about which universe you’re really in—your consciousness itself defines (encompasses?) which universes contain you. So, when your memories become more coarse, you’re not just becoming uncertain about which universe you’re in—you’re changing which universes count as containing you, since your consciousness is the only arbiter of this.
A cool way to measure dishonesty: How many people claim to have completed an impossible five minute task.
This has since been community noted, fairly from my understanding.
This graph is not about how many people reported completing a task in 5 minutes when that was not true, this graph shows how many people completed the whole task even though it took them more than 5 minutes (which was all the time they were getting paid for).
Derek Lowe I believe does the closest to a Matt Levine for Pharma (and chem): https://www.science.org/blogs/pipeline
He has a really fun to read series titled “Things I Won’t Work With” where he talks a bunch about dangerous chemicals: https://www.science.org/topic/blog-category/things-i-wont-work-with
I found this to be a valuable post!
I disagree with your conclusion though—the thoughts that come to my mind as to why are:
You seem overly anchored on COT as the only scaffolding system in the near-mid future (2-5 years). While I’m uncertain what specific architectures will emerge, the space of possible augmentations (memory systems, tool use, multi-agent interactions, etc.) seems vastly larger than current COT implementations.
Your crux that “LLMs have never done anything important” feels only mildly compelling. Anecdotally, many people do feel LLM’s significantly improve their ability to do important and productive work, both work which requires creativity/cross field information integration and work that does not.
Further, I am not aware of any large scale ($10 million+) instances of people trying something like a better version of “Ask an LLM to list out in context fields it feels like would be ripe for information integration leading to a breakthrough, and then do further reasoning on what those breakthroughs are/actually perform them.”
Something like that seems like it would be a MVP of “actually try and get an LLM to come up with something significantly economically valuable. I expect that the lack of this type of experiment existing is because major AI labs feel like that would be choosing to exploit while there are still many gains to be made from exploring further architectural and scaffolding-esque improvements.
Where you say “Certainly LLMs should be useful tools for coding, but perhaps not in a qualitatively different way than the internet is a useful tool for coding, and the internet didn’t rapidly set off a singularity in coding speed.”, I find this to be untrue both in terms of the impact of the internet (while it did not cause a short takeoff, it did dramatically increase the amount of new programmers and the effective transfer of information between them. I expect without it we would see computers having <20% of their current economic impact), and in terms of the current and expected future impact of LLM’s (LLM’s simply are widely used by smart/capable programmers. I trust them to evaluate if it is noticeably better than StackOverflow/the rest of the internet).