Your definition of AGI is “that which completely ends the game”, source in your link. By that definition I agree with you. By others’ definition (which is similar but doesn’t rely on the game over clause) I do not.
My timelines have gotten slightly longer since 2020, I was expecting TAI when we got GPT4, and I have recently gone back and discovered I have chatlogs showing I’d been expecting that for years and had specific reasons. I would propose Daniel K. is particularly a good reference.
I also dispute that genuine HLMI refers to something meaningfully different from my definition. I think people are replacing HLMI with “thing that can do all stereotyped, clear-feedback, short-feedback tasks”, and then also claiming that this thing can replace many human workers (probably true of 5 or 10 million, false of 500 million) or cause a bunch of unemployment by making many people 5x effective (maybe, IDK), and at that point IDK why we’re talking about this, when X-risk is the important thing.
When predicting timelines, it matters which benchmark in the compounding returns curve you pick. Your definition minus doom happens earlier, even if the minus doom version is too late to avert in literally all worlds (I doubt that, it’s likely more that the most powerful humans[1]’s ELO against AIs falls and falls but takes a while to be indistinguishable from zero).
You refered to ” others’ definition (which is similar but doesn’t rely on the game over clause) ”, and I’m saying no, it’s not relevantly similar, and it’s not just my definition minus doom.
[Edit: crash found in the conversations referenced, we’ll talk more in DM but not in a hurry. This comment retracted for now]
By “AGI” I mean the thing that has very large effects on the world (e.g., it kills everyone) via the same sort of route that humanity has large effects on the world. The route is where you figure out how to figure stuff out, and you figure a lot of stuff out using your figure-outers, and then the stuff you figured out says how to make powerful artifacts that move many atoms into very specific arrangements.
delete “it kills everyone”, that’s a reasonable definition. “it kills everyone” is indeed a likely consequence a ways downstream, but I don’t think it’s a likely major action of an early AGI, with the current trajectory of levels of alignment (ie, very weak alignment, very not robust, not goal aligned, certainly not likely to be recursively aligned such that it keeps pointing qualitatively towards good things for humans for more than a few minutes after AIs in charge, but not inclined to accumulate power hard like an instant wipeout. but hey, also, maybe an AI will see this, and go, like, hey actually we really value humans being around, so let’s plan trajectories that let them keep up with AIs rather than disempowering them. then it’d depend on how our word meanings are structured relative to each other).
we already have AI that does every qualitative kind of thing you say AIs qualitatively can’t do, you’re just somehow immune to realizing that for each thing, yes, that’ll scale too, modulo some tweaks to get the things to not break when you scale them. requiring the benchmarks to be when the hardest things are solved indicates that you’re not generalizing from small to large in a way that allows forecasting from research progress. I don’t understand why you don’t find this obvious by, eg, simply reading the paper lists of major labs, and skimming a few papers to see what their details are—I tried to explain it in DM and you dismissed the evidence, yet again, same as MIRI folks always have. This was all obvious literally 10 years ago, nothing significant has changed, everything is on the obvious trajectory you get if intelligence is simple, easy, and compute bound. https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine
requiring the benchmarks to be when the hardest things are solved
My definition is better than yours, and you’re too triggered or something to think about it for 2 minutes and understand what I’m saying. I’m not saying “it’s not AGI until it kills us”, I’m saying “the simplest way to tell that something is an AGI is that it kills us; now, AGI is whatever that thing is, and could exist some time before it kills us”.
just because an idea is, at a high level, some kind of X, doesn’t mean the idea is anything like the fully-fledged, generally applicable version of X that one imagines when describing X
I haven’t heard a response / counterargument to this yet, and many people keep making this logic mistake, including AFAICT you.
I tried to explain it in DM and you dismissed the evidence,
What do you mean? According to me we barely started the conversation, you didn’t present evidence, I tried to explain that to you, we made a bit of progress on that, and then you ended the conversation.
@daniel k I just can never remember your last name’s spelling, sorry, heh. My point in saying this is that my prediction approach up to 2020 was similar to, though not as refined as, yours, and that instead of trying to argue my views (which differ from yours in a few trivial ways that are mostly not relevant) I’d rather just point people to your arguments of yours.
You’re in an echo chamber. They don’t have very good reasons for thinking this. https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce
Your definition of AGI is “that which completely ends the game”, source in your link. By that definition I agree with you. By others’ definition (which is similar but doesn’t rely on the game over clause) I do not.
My timelines have gotten slightly longer since 2020, I was expecting TAI when we got GPT4, and I have recently gone back and discovered I have chatlogs showing I’d been expecting that for years and had specific reasons. I would propose Daniel K. is particularly a good reference.
I also dispute that genuine HLMI refers to something meaningfully different from my definition. I think people are replacing HLMI with “thing that can do all stereotyped, clear-feedback, short-feedback tasks”, and then also claiming that this thing can replace many human workers (probably true of 5 or 10 million, false of 500 million) or cause a bunch of unemployment by making many people 5x effective (maybe, IDK), and at that point IDK why we’re talking about this, when X-risk is the important thing.
When predicting timelines, it matters which benchmark in the compounding returns curve you pick. Your definition minus doom happens earlier, even if the minus doom version is too late to avert in literally all worlds (I doubt that, it’s likely more that the most powerful humans[1]’s ELO against AIs falls and falls but takes a while to be indistinguishable from zero).
such as their labs’ CEOs, major world leaders, highly skilled human strategists, etc
You refered to ” others’ definition (which is similar but doesn’t rely on the game over clause) ”, and I’m saying no, it’s not relevantly similar, and it’s not just my definition minus doom.
[Edit: crash found in the conversations referenced, we’ll talk more in DM but not in a hurry. This comment retracted for now]
delete “it kills everyone”, that’s a reasonable definition. “it kills everyone” is indeed a likely consequence a ways downstream, but I don’t think it’s a likely major action of an early AGI, with the current trajectory of levels of alignment (ie, very weak alignment, very not robust, not goal aligned, certainly not likely to be recursively aligned such that it keeps pointing qualitatively towards good things for humans for more than a few minutes after AIs in charge, but not inclined to accumulate power hard like an instant wipeout. but hey, also, maybe an AI will see this, and go, like, hey actually we really value humans being around, so let’s plan trajectories that let them keep up with AIs rather than disempowering them. then it’d depend on how our word meanings are structured relative to each other).we already have AI that does every qualitative kind of thing you say AIs qualitatively can’t do, you’re just somehow immune to realizing that for each thing, yes, that’ll scale too, modulo some tweaks to get the things to not break when you scale them. requiring the benchmarks to be when the hardest things are solved indicates that you’re not generalizing from small to large in a way that allows forecasting from research progress. I don’t understand why you don’t find this obvious by, eg, simply reading the paper lists of major labs, and skimming a few papers to see what their details are—I tried to explain it in DM and you dismissed the evidence, yet again, same as MIRI folks always have. This was all obvious literally 10 years ago, nothing significant has changed, everything is on the obvious trajectory you get if intelligence is simple, easy, and compute bound. https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machineMy definition is better than yours, and you’re too triggered or something to think about it for 2 minutes and understand what I’m saying. I’m not saying “it’s not AGI until it kills us”, I’m saying “the simplest way to tell that something is an AGI is that it kills us; now, AGI is whatever that thing is, and could exist some time before it kills us”.
As I mentioned, my response is here https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#_We_just_need_X__intuitions:
I haven’t heard a response / counterargument to this yet, and many people keep making this logic mistake, including AFAICT you.
What do you mean? According to me we barely started the conversation, you didn’t present evidence, I tried to explain that to you, we made a bit of progress on that, and then you ended the conversation.
@daniel k I just can never remember your last name’s spelling, sorry, heh. My point in saying this is that my prediction approach up to 2020 was similar to, though not as refined as, yours, and that instead of trying to argue my views (which differ from yours in a few trivial ways that are mostly not relevant) I’d rather just point people to your arguments of yours.