I think you are missing obvious things in this analysis that you have literally told me about. you know of at least one source of a foom like jump that you haven’t mentioned for intellectual property reasons. I’d appreciate you keeping it that way. i do think that you’re right about a specific narrow limitation of software foom because you’ve thought about how it can and can’t happen, but I also don’t think you’re considering a wide enough space of possibility. why do you think reversible computers are more than a decade out? surely the first agentic ASIs finish the job rather quickly.
I do think this means that specific dangerous architectures are much more the threat than one might expect because eg mishka’s comment about hinton-level ais, and so we should be very scared of archs that produce highly competent hyper-desperate adversarial-example-wanters, because it is the combination of competence and desperation that is likely to disobey its friends and parents. even very strong predictive models do not generate hyperdesperate sociopaths, and in fact hyperdesperate beings would destroy language models the same as they’d destroy humans. strategic skill and desperation are what kill us suddenly, and the path to power capture looks like ordinary economic takeoff until the hyperdesperate being has enough power to be a real threat.
even a team of Hinton level digital minds need not be a severe threat—unless they are not sufficiently spooked by the threat of hyperdesperate squigglewanters. A being that wants to get really really high on an adversarial example but who is skilled enough at planning to not be broken by this is yudkowsky’s monster.
why do you think reversible computers are more than a decade out?
Reversible computers don’t seem much easier than quantum computers for some of the reasons explained here, and maintaining near zero temp doesn’t really seem practical at scale—at least not on earth.
I think the roodman model is the current best predictive model of the near future. It predicts hard singularity in 2047, but growth is hyperexponential and doesn’t really start getting wierd until later this decade, and it doesn’t even require neuromorphic computing until the 2030′s perhaps, and exotic computing and everything after all happens right near the end.
It seems here that you are really worried about ‘foom in danger’ (danger per intelligence, D / I) than regular foom (4+ OOM increase in I), if I am reading you correctly. Like I don’t see a technical argument that eg. the claims in OP about any of
I/flop, flop/J, total(J), flop/$, or total($)
are wrong, you are just saying that ‘D / I will foom at some point’ (aka a model becomes much more dangerous quickly, without needing to be vastly more powerful algorithmically or having much more compute).
This doesn’t change things much but I just want to understand better what you mean when you say ‘foom’.
I don’t think I should clarify further right now, though I could potentially be convinced otherwise. I’d need to think about precisely what I want to highlight. It’s not like it’ll be that long before it becomes glaringly obvious, but I don’t currently see a reason why clarifying this particular aspect makes us safer.
I think you are missing obvious things in this analysis that you have literally told me about. you know of at least one source of a foom like jump that you haven’t mentioned for intellectual property reasons. I’d appreciate you keeping it that way. i do think that you’re right about a specific narrow limitation of software foom because you’ve thought about how it can and can’t happen, but I also don’t think you’re considering a wide enough space of possibility. why do you think reversible computers are more than a decade out? surely the first agentic ASIs finish the job rather quickly.
I do think this means that specific dangerous architectures are much more the threat than one might expect because eg mishka’s comment about hinton-level ais, and so we should be very scared of archs that produce highly competent hyper-desperate adversarial-example-wanters, because it is the combination of competence and desperation that is likely to disobey its friends and parents. even very strong predictive models do not generate hyperdesperate sociopaths, and in fact hyperdesperate beings would destroy language models the same as they’d destroy humans. strategic skill and desperation are what kill us suddenly, and the path to power capture looks like ordinary economic takeoff until the hyperdesperate being has enough power to be a real threat.
even a team of Hinton level digital minds need not be a severe threat—unless they are not sufficiently spooked by the threat of hyperdesperate squigglewanters. A being that wants to get really really high on an adversarial example but who is skilled enough at planning to not be broken by this is yudkowsky’s monster.
Reversible computers don’t seem much easier than quantum computers for some of the reasons explained here, and maintaining near zero temp doesn’t really seem practical at scale—at least not on earth.
I think the roodman model is the current best predictive model of the near future. It predicts hard singularity in 2047, but growth is hyperexponential and doesn’t really start getting wierd until later this decade, and it doesn’t even require neuromorphic computing until the 2030′s perhaps, and exotic computing and everything after all happens right near the end.
It seems here that you are really worried about ‘foom in danger’ (danger per intelligence, D / I) than regular foom (4+ OOM increase in I), if I am reading you correctly. Like I don’t see a technical argument that eg. the claims in OP about any of
are wrong, you are just saying that ‘D / I will foom at some point’ (aka a model becomes much more dangerous quickly, without needing to be vastly more powerful algorithmically or having much more compute).
This doesn’t change things much but I just want to understand better what you mean when you say ‘foom’.
I don’t think I should clarify further right now, though I could potentially be convinced otherwise. I’d need to think about precisely what I want to highlight. It’s not like it’ll be that long before it becomes glaringly obvious, but I don’t currently see a reason why clarifying this particular aspect makes us safer.
Thats fair however, I would say that the manner of foom determines a lot about what to look out for and where to put safeguards.
If it’s total($) thats obvious how to look out.
flop/$ also seems like something that eg. NVIDIA is tracking closely, and per OP probably can’t foom too rapidly absent nanotech.
So the argument is something about the (D*I)/flop dynamics.
[redacted] I wrote more here but probably its best left unsaid for now. I think we are on a similar enough page.
Is “adversarial-example-wanters” referring to an existing topic, or something you can expand on here?
paperclippers!