garrison

Karma: 1,246

garrison Apr 24, 2025, 8:48 PM
2 points
0
in reply to: thedudeabides’s comment on: Why Were We Wrong About China and AI? A Case Study in Failed Rationality
Sorry, lot on my plate.
You’re basically asking how we’d operationalize the claim that either the USG or PRC are “racing toward AGI”? Probably would involve some dramatic action like consolidating large amounts of compute into projects that are either nationalized or contracted to the govt (like this part of AI-2027:
“A Centralized Development Zone (CDZ) is created at the Tianwan Power Plant (the largest nuclear power plant in the world) to house a new mega-datacenter for DeepCent, along with highly secure living and office spaces to which researchers will eventually relocate. Almost 50% of China’s AI-relevant compute is now working for the DeepCent-led collective,38 and over 80% of new chips are directed to the CDZ.39 At this point, the CDZ has the power capacity in place for what would be the largest centralized cluster in the world.”
Do you want to suggest specific thresholds or modifications?

garrison Mar 24, 2025, 4:51 PM
8 points
6
on: Why Were We Wrong About China and AI? A Case Study in Failed Rationality
I don’t think it’s fair to say I made a bad prediction here.
Here’s the full context of my quote: “The report clocks in at a cool 793 pages with 344 endnotes. Despite this length, there are only a handful of mentions of AGI, and all of them are in the sections recommending that the US race to build it.
In other words, there is no evidence in the report to support Helberg’s claim that “China is racing towards AGI.”
Nonetheless, his quote goes unchallenged into the 300-word Reuters story, which will be read far more than the 800-page document. It has the added gravitas of coming from one of the commissioners behind such a gargantuan report.
I’m not asserting that China is definitively NOT rushing to build AGI. But if there were solid evidence behind Helberg’s claim, why didn’t it make it into the report?”
Here’s my tweet mentioning Gwern’s comment. It’s not clear that DeepSeek falsifies what Gwern said here:
1. the scientific culture of China is ‘mafia’ like (Hsu’s term, not mine) and focused on legible easily-cited incremental research, and is against making any daring research leaps or controversial breakthroughs...
  but is capable of extremely high quality world-class followup and large scientific investments given a clear objective target and government marching orders
V3 and R1 are impressive but didn’t advance the absolute capabilities frontier. Maybe the capabilities/cost frontier, though we don’t actually know how compute efficient OAI, Anthropic, GDM are.
I think this part of @gwern’s comment doesn’t hold up as well:
2. there is no interest or investment in an AI arms race, in part because of a “quiet confidence” (ie. apathy/lying-flat) that if anything important happens, fast-follower China can just catch up a few years later and win the real race. They just aren’t doing it. There is no Chinese Manhattan Project. There is no race. They aren’t dumping the money into it, and other things, like chips and Taiwan and demographics, are the big concerns which have the focus from the top of the government, and no one is interested in sticking their necks out for wacky things like ‘spending a billion dollars on a single training run’ without explicit enthusiastic endorsement from the very top.
I still don’t think DS is evidence that “China” is racing toward AGI. The US isn’t racing toward AGI either. Some American companies are, with varying levels of support from the government. But there’s a huge gap between that and Manhattan Project levels of direct govt investment, support, and control.
However, overall, I do think that DS has gotten the CCP more interested in AGI and changed the landscape a lot.

garrison Feb 11, 2025, 5:04 PM
1 point
0
in reply to: rvnnt’s comment on: Why Did Elon Musk Just Offer to Buy Control of OpenAI for $100 Billion?
Fixed, thanks!

garrison Jan 16, 2025, 5:49 PM
1 point
0
in reply to: _liminaldrift’s comment on: Is AI Hitting a Wall or Moving Faster Than Ever?
I think that is a problem for the industry, but probably not an insurmountable barrier the way some commentators make it out to be.
1. o-series of models may be able to produce new high quality training data
2. sufficiently good reasoning approaches + existing base models + scaffolding may be sufficient to get you to automating ML research
One other thought is that there’s probably an upper limit on how good an LLM can get even with unlimited high quality data and I’d guess that models would asymptotically approach it for a while. Based on the reporting around GPT-5 and other next-gen models, I’d guess that the issue is lack of data rather than approaching some fundamental limit.

garrison Dec 23, 2024, 8:10 PM
4 points
0
in reply to: Edy Nastase’s comment on: We are in a New Paradigm of AI Progress—OpenAI’s o3 model makes huge gains on the toughest AI benchmarks in the world
It was all my twitter feed was talking about, but I think it’s been really under-discussed in mainstream press.
RE Knoop’s comment, here are some relevant grafs from the ARC announcement blog post:
To adapt to novelty, you need two things. First, you need knowledge – a set of reusable functions or programs to draw upon. LLMs have more than enough of that. Second, you need the ability to recombine these functions into a brand new program when facing a new task – a program that models the task at hand. Program synthesis. LLMs have long lacked this feature. The o series of models fixes that.
For now, we can only speculate about the exact specifics of how o3 works. But o3′s core mechanism appears to be natural language program search and execution within token space – at test time, the model searches over the space of possible Chains of Thought (CoTs) describing the steps required to solve the task, in a fashion perhaps not too dissimilar to AlphaZero-style Monte-Carlo tree search. In the case of o3, the search is presumably guided by some kind of evaluator model. To note, Demis Hassabis hinted back in a June 2023 interview that DeepMind had been researching this very idea – this line of work has been a long time coming.
More in the ARC post.
My rough understanding is that it’s like a meta-CoT strategy, evaluating multiple different approaches.

garrison Nov 28, 2024, 7:21 PM
9 points
1
in reply to: Adrian Garcia’s comment on: China Hawks are Manufacturing an AI Arms Race
This is what Hsu just said about it: “3. I could be described as a China hawk in that I’ve been pointing to a US-China competition as unavoidable for over a decade. But I think I have more realistic views about what is happening in PRC than most China hawks. I also try to focus on simple descriptive analysis rather than getting distracted by normative midwit stuff.”
https://x.com/hsu_steve/status/1861970671527510378

garrison Nov 28, 2024, 7:21 PM
11 points
2
in reply to: gwern’s comment on: China Hawks are Manufacturing an AI Arms Race
Steve Hsu clarified some things on my thread about this discussion: https://x.com/hsu_steve/status/1861970671527510378
“Clarifications:
1. The mafia tendencies (careerist groups working together out of self-interest and not to advance science itself) are present in the West as well these days. In fact the term was first used in this way by Italian academics.
2. They’re not against big breakthroughs in PRC, esp. obvious ones. The bureaucracy bases promotions, raises, etc. on metrics like publications in top journals, cititations, … However there are very obvious wins that they will go after in a coordinated way—including AI, semiconductors, new energy tech, etc.
3. I could be described as a China hawk in that I’ve been pointing to a US-China competition as unavoidable for over a decade. But I think I have more realistic views about what is happening in PRC than most China hawks. I also try to focus on simple descriptive analysis rather than getting distracted by normative midwit stuff.
4. There is coordinated planning btw govt and industry in PRC to stay at the frontier in AI/AGI/ASI. They are less susceptible to “visionaries” (ie grifters) so you’ll find fewer doomers or singularitarians, etc. Certainly not in the top govt positions. The quiet confidence I mentioned extends to AI, not just semiconductors and other key technologies.”

garrison Nov 21, 2024, 11:52 PM
1 point
0
in reply to: Raemon’s comment on: China Hawks are Manufacturing an AI Arms Race
Gotcha, well I’m on it!

garrison Nov 21, 2024, 11:51 PM
4 points
1
in reply to: Logan Zoellner’s comment on: China Hawks are Manufacturing an AI Arms Race
Interesting, do you have a link for that?
US companies are racing toward AGI but the USG isn’t. As someone else mentioned, Dylan Patel from Semianalysis does not think China is scale-pilled.

garrison Nov 21, 2024, 4:18 AM
11 points
2
in reply to: Raemon’s comment on: China Hawks are Manufacturing an AI Arms Race
As mentioned in another reply, I’m planning to do a lot more research and interviews on this topic, especially with people who are more hawkish on China. I also think it’s important that unsupported claims with large stakes get timely pushback, which is in tension with the type of information gathering you’re recommending (which is also really important, TBC!).

garrison Nov 21, 2024, 4:15 AM
4 points
1
in reply to: Jacob Pfau’s comment on: China Hawks are Manufacturing an AI Arms Race
Claiming that China as a country is racing toward AGI != Chinese AI companies aren’t fast following US AI companies, which are explicitly trying to build AGI. This is a big distinction!

garrison Nov 21, 2024, 4:12 AM
10 points
6
in reply to: Seth Herd’s comment on: China Hawks are Manufacturing an AI Arms Race
Hey Seth, appreciate the detailed engagement. I don’t think the 2017 report is the best way to understand what China’s intentions are WRT to AI, but there was nothing in the report to support Helberg’s claim to Reuters. I also cite multiple other sources discussing more recent developments (with the caveat in the piece that they should be taken with a grain of salt). I think the fact that this commission was not able to find evidence for the “China is racing to AGI” claim is actually pretty convincing evidence in itself. I’m very interested in better understanding China’s intentions here and plan to deep dive into it over the next few months, but I didn’t want to wait until I could exhaustively search for the evidence that the report should have offered while an extremely dangerous and unsupported narrative takes off.

I also really don’t get the error pushback. These really were less technical errors than basic factual errors and incoherent statements. They speak to a sloppiness that should affect how seriously the report should be taken. I’m not one to gatekeep ai expertise, but idt it’s too much to expect a congressional commission with a top recommendation to commence in a militaristic AI arms race to have SOMEONE read a draft who knows that chatgpt-3 isn’t a thing.

garrison Nov 14, 2024, 4:16 PM
1 point
0
in reply to: Vladimir_Nesov’s comment on: Is Deep Learning Actually Hitting a Wall? Evaluating Ilya Sutskever’s Recent Claims
Thanks for these!

garrison Nov 13, 2024, 10:10 PM
6 points
0
in reply to: Noosphere89’s comment on: Is Deep Learning Actually Hitting a Wall? Evaluating Ilya Sutskever’s Recent Claims
I think this is a misunderstanding of the piece and how journalists typically paraphrase things. The reporters wrote that Ilya told them that results from scaling up pre-training have plateaued. So he probably said something to that effect, but for readability and word-count reasons, they paraphrased it.
If a reported story from a credible outlet says something like X told us that Y, then the reporters are sourcing claim Y to X, whether or not they include a direct quote.
The plateau claim also jives with The Information story about OpenAI, as well as a few other similar claims made by people in industry.
Ilya probably spoke to the reporter(s) for at least a few min, so the quotes you see are a tiny fraction of everything he said.

garrison Jul 17, 2024, 3:40 PM
9 points
0
in reply to: Ruby’s comment on: Against Aschenbrenner: How ‘Situational Awareness’ constructs a narrative that undermines safety and threatens humanity
FWIW I was also confused by this usage of sic, bc I’ve only ever seen it as indicating the error was in the original quote. Quotes seem sufficient to indicate you’re quoting the original piece. I use single quotes when I’m not quoting a specific person, but introducing a hypothetical perspective.

garrison Mar 28, 2024, 10:38 PM
2 points
0
in reply to: trevor’s comment on: My Interview With Cade Metz on His Reporting About Slate Star Codex
I only skimmed the NYT piece about China and ai talent, but didn’t see evidence of what you said (dishonestly angle shooting the AI safety scene).

garrison Mar 5, 2024, 6:13 PM
1 point
0
in reply to: Nathan Helm-Burger’s comment on: Claude Doesn’t Want to Die
The fey thing stuck out to me too. I’ll guess ChatGPT?
I agree that it’s hard to disentangle the author/character thing. I’m really curious for what the base model would say about its situation (especially without the upstream prompt “You are a language model developed by...”).

garrison Feb 14, 2024, 3:18 AM
2 points
0
in reply to: brambleboy’s comment on: My cover story in Jacobin on AI capitalism and the x-risk debates
Thank you so much! I haven’t gotten any serious negative feedback from lefties for the EA stuff so far, though an e/acc on Twitter mentioned it haha

garrison Feb 12, 2024, 9:56 PM
2 points
1
in reply to: Logan Zoellner’s comment on: Sam Altman’s Chip Ambitions Undercut OpenAI’s Safety Strategy
Maybe I wasn’t clear enough in the writing, but I make basically the same point about the desirability of a slow takeoff in the piece.

garrison Feb 10, 2024, 7:56 PM
13 points
20
on: OpenAI wants to raise 5-7 trillion
This approach appears to directly contradict Altman’s blogpost from less than a year ago arguing for short timelines + slow takeoff because of less compute overhang. I wrote more on this here.