Greg C

Karma: 243

Greg C Apr 22, 2025, 2:21 PM
1 point
0
in reply to: mishka’s comment on: How We Might All Die in A Year
then it would be better to use an example not directly aimed against “our atoms”
All the atoms are getting repurposed at once, no special focus on those in our bodies (but there is in the story, to get the reader to empathise). Maybe I could’ve included more description of non-alive things getting destroyed.
mucking with quantum gravity too recklessly, or smth in that spirit
I’m trying to focus on plausible science/tech here.
they need to do experiments in forming hybrid consciousness with humans to crack the mystery of human subjectivity, to experience that first-hand for themselves, and to decide whether that is of any value to them based on the first-hand empirical material (losing that option without looking is a huge loss)
Interesting. But even if they do find something valuable in doing that, there’s not much to keep the vast majority of humans around. And as you say, they could just end up as “scans”, with very few being run as oracles.

Greg C Mar 31, 2025, 10:19 AM
1 point
0
in reply to: mishka’s comment on: How We Might All Die in A Year
Where does my writing suggest that it’s a “power play” and “us vs them”? (That was not the intention at all! I’ve always seen indifference, and “collateral damage” as the biggest part of ASI x-risk.)
as we know, compute is not everything, algorithmic improvement is even more important
It should go without saying that it would also be continually improving it’s algorithms. But maybe I should’ve made that explicit.
the action the ASI is taking in the OP is very suboptimal and deprives it of all kinds of options
What are some examples of these options?

Greg C Mar 31, 2025, 10:14 AM
1 point
0
in reply to: Kabir Kumar’s comment on: How We Might All Die in A Year
They don’t have a choice in the matter—it’s forced by the government (nationalisation). This kind of thing has happened before in wartime (without the companies or people involved staging a rebellion).

Greg C Mar 29, 2025, 11:02 AM
1 point
0
in reply to: mishka’s comment on: How We Might All Die in A Year
On one hand, it’s not clear if a system needs to be all that super-smart to design a devastating attack of this kind...
Good point, but—and as per your second point too—this isn’t an “attack”, it’s “go[ing] straight for execution on its primary instrumental goal of maximally increasing its compute scaling” (i.e. humanity and biological life dying is just collateral damage).
probably would not want to irreversibly destroy important information without good reasons
Maybe it doesn’t consider the lives of individual organisms as “important information”? But if it did, it might do something like scan as it destroys, to retain the information content.

Greg C Mar 22, 2025, 10:33 PM
2 points
0
on: Why Were We Wrong About China and AI? A Case Study in Failed Rationality
Are you saying they are suicidal?

Greg C May 4, 2023, 8:08 AM
5 points
0
on: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
LessWrong:

A post about all the reasons AGI will kill us: No. 1 all time highest karma (827 on 467 votes; +1.77 karma/vote)
A post about containment strategy for AGI: 7th all time highest karma (609 on 308 votes; +1.98 karma/vote)
A post about us all basically being 100% dead from AGI: 52nd all time highest karma (334 on 343 votes; +0.97 karma/vote, a bit more controversial)

Also LessWrong:

A post about actually doing something about containing the threat from AGI and not dying [this one]: downvoted to oblivion (-5 karma within an hour; currently 13 karma on 24 votes; +0.54 karma/vote)

My read: y’all are so allergic to anything considered remotely political (even though this should really not be a mater of polarisation—it’s about survival above all else!) that you’d rather just lie down and be paperclipped than actually do anything to prevent it happening. I’m done.

Greg C May 4, 2023, 7:31 AM
3 points
0
in reply to: 1a3orn’s comment on: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
From the Abstract:
Rather than targeting state-of-the-art performance, our objective is to highlight GPT-4’s potential
They weren’t aiming for SOTA! What happens when they do?

Greg C May 4, 2023, 7:27 AM
8 points
0
in reply to: 1a3orn’s comment on: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
The way I see the above post (and it’s accompaniment) is knocking down all the soldiers that I’ve encountered talking to lots of people about this over the last few weeks. I would appreciate it if you could stand them back up (because I’m really trying to not be so doomy, and not getting any satisfactory rebuttals).

Greg C May 4, 2023, 7:23 AM
6 points
1
in reply to: Ulisse Mini’s comment on: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Thanks for writing out your thoughts in some detail here. What I’m trying to say is that things are already really bad. Industry self-regulation has failed. At some point you have to give up on hoping that the fossil fuel industry (AI/ML industry) will do anything more to fix climate change (AGI x-risk) than mere greenwashing (safetywashing). How much worse does it need to get for more people to realise this?

The Alignment community (climate scientists) can keep doing their thing; I’m very much in favour of that. But there is also now an AI Notkilleveryoneism (climate action) movement. We are raising the damn Fire Alarm.

From the post you link:
some authority somewhere will take notice and come to the rescue.
Who is that authority?
The United Nations Security Council. Anything less and we’re toast.
And we can talk all we like about the unilateralist’s curse, but I don’t think anything a bunch of activists can do will ever top the formation and corruption-to-profit-seeking of OpenAI and Anthropic (the supposedly high status moves).

Greg C May 3, 2023, 10:59 PM
15 points
4
in reply to: 1a3orn’s comment on: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
It’s really not intended as a gish gallop, sorry if you are seeing it as such. I feel like I’m really only making 3 arguments:

1. AGI is near
2. Alignment isn’t ready (and therefore P(doom|AGI is high)
3. AGI is dangerous
And then drawing the conclusion from all these that we need a global AGI moratorium asap.

Greg C May 3, 2023, 10:53 PM
9 points
2
in reply to: 1a3orn’s comment on: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
I think you need to zoom out a bit and look at the implications of these papers. The danger isn’t in what people are doing now, it’s in what they might be doing in a few months following on from this work. The NAS paper was a proof of concept. What happens when it’s massively scaled up? What happens when efficiency gains translate into further efficiency gains?

Greg C Apr 25, 2023, 4:20 PM
10 points
2
on: It’s time for EA leadership to pull the fast-takeoff fire alarm.
This post was only a little ahead of it’s time. The time is now. EA/LW will probably be eclipsed by wider public campaigning on this if they (the leadership) don’t get involved.

Greg C Apr 16, 2023, 9:09 AM
5 points
2
in reply to: Adam Kaufman ’s comment on: A freshman year during the AI midgame: my approach to the next year
Advocate for a global moratorium on AGI. Try and buy (us all) more time. Learn the basics of AGI safety (e.g. AGI Safety Fundamentals) so you are able to discuss the reasons why we need a moratorium in detail. YMMV, but this is what I’m doing as a financially independent 42 year-old. I feel increasingly like all my other work is basically just rearranging deckchairs on the Titanic.

Greg C Apr 13, 2023, 11:31 AM
2 points
−1
on: Campaign for AI Safety: Please join me
Thank you for doing this. I’m thinking that at this point, there needs to be an organisation with the singular goal of pushing for a global moratorium on AGI development. Anyone else interested in this? Have DM’d.

Greg C Mar 31, 2023, 1:27 PM
1 point
0
in reply to: Gerald Monroe’s comment on: FLI open letter: Pause giant AI experiments
1. Ok, I admit I simplified here. There is still probably ~ a million times (give or take an order of magnitude) more relevant compute (GPUs, TPUs) than was used to train GPT-4.
2. It won’t need large orders to gain a relevant foothold. Just a few tiny orders could suffice.
3. I didn’t mean literallly rob the stock market. I’m referring to out-trading all the other traders (inc. existing HFT) to accumulate resources.
4. Exponential growth can’t remain “slow” forever, by definition. How long does it take for the pond to be completely covered by lily pads when it’s half covered? How long did it take for Covid to become a pandemic? Not decades.
5. I referred to social hacking (i.e. blackmailing people into giving up their passwords). This could go far enough (say, at least 10% of world devices). Maybe quantum computers (or some better tech the AI thinks up) could do the rest.

Greg C Mar 31, 2023, 10:05 AM
14 points
5
on: Pausing AI Developments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky
Is this now on the radar of national security agencies and the UN Security Council? Is it being properly discussed inside the US government? If not, are meetings being set up? Would be good if someone in the know could give an indication (I hope Yudkowsky is busy talking to lots of important people!)

Greg C Mar 31, 2023, 7:06 AM
1 point
0
in reply to: Gerald Monroe’s comment on: FLI open letter: Pause giant AI experiments
Can you be more specific about what you don’t agree with? Which parts can’t happen, and why?

Greg C Mar 30, 2023, 9:09 PM
1 point
0
in reply to: Gerald Monroe’s comment on: FLI open letter: Pause giant AI experiments
Compute—what fraction of world compute did it take to train GPT-4? Maybe 1e-6? There’s 1e6 improvement right there from a superhuman GPT-6 capturing all of the “hardware overhang”.
Data—superhuman GPT-6 doesn’t need to rely on human recorded data, it can harness all the sensors on the planet to gather exabytes of-real time data per second, and re-derive scientific theories from scratch in minutes based on it’s observations (including theories about human behaviour, language etc)

Robotics/Money—easy for GPT-6. Money it can get from scamming gullible humans, hacking crypto wallets via phishing/ransomware, or running rings round stock market traders. Robotics it can re-derive and improve on from it’s real time sensing of the planet and it’s speed of thought making our daily life look like geology does to us. It can escape to the physical world any number of ways by manipulating humans into giving it access to boot loaders for it to gain a foothold in the physical world (robots, mail-order DNA etc).

Algorithm search time—wall clock time is much reduced when you’ve just swallowed the world’s hardware overhang (see Compute above)

Factoring the above, your extra decades become extra hours.

Greg C Mar 30, 2023, 7:56 AM
2 points
1
in reply to: Gerald Monroe’s comment on: FLI open letter: Pause giant AI experiments
Selection pressure will cause models to become agentic as they increase in power—those doing the agentic things (following universal instrumental goals like accumulating more resources and self-improvement) will outperform those that don’t. Mesaoptimisation (explainer video) is kind of like cheating—models that create inner optimisers that target something easier to get than what we meant, will be selected (by getting higher rewards) over models that don’t (because we won’t be aware of the inner misalignment). Evolution is a case in point—we are products of it, yet misaligned to its goals (we want sex, and high calorie foods, and money, rather than caring explicitly about inclusive genetic fitness). Without alignment being 100% watertight, powerful AIs will have completely alien goals.

Greg C Mar 30, 2023, 7:32 AM
1 point
0
in reply to: Noosphere89’s comment on: FLI open letter: Pause giant AI experiments
Yeah, they work well enough at this (~human) level. But no current alignment techniques are scalable to superhuman AI. I’m worried that basically all of the doom flows through an asymptote of imperfect alignment. I can’t see how this doesn’t happen, short of some “miracle”.