Hmm, I guess it’s just really not obvious that your proposed bet here disagrees with the OP. I think I roughly believe both the things that the OP says, and also wouldn’t take your bet. It still feels like a fine bet to offer, but I am confused why it’s phrased so much in contrast to the OP. If you are confident we are not going to see large dangerous capability gains in the next 5-7 years, I think I would much prefer you make a bet that tries to offer corresponding odds and the specific capability gains (though that runs a bit into “betting on doomsday” problems)
If you are confident we are not going to see large dangerous capability gains in the next 5-7 years, I think I would much prefer you make a bet that tries to offer corresponding odds and the specific capability gains (though that runs a bit into “betting on doomsday” problems)
What are the “specific cabability gains” you are referring to? I don’t see any specific claims in the post we are responding to. By contrast, we listed 7 concrete tasks that appear trivial to perform if we are AGI-levels of capability, and very easy if we are only a few steps from AGI. I’d be genuinely baffled if you think AGI can be imminent at the same time we still don’t have good self-driving cars, robots that can wash dishes, or AI capable of doing well on mathematics word problems. This view would seem to imply that we will get AGI pretty much out of nowhere.
An AGI might become a dictator in every country on earth while still not being able to wash dishes or make errors when it comes to driving 100,000 miles. Physical coordination is not required.
It’s not clear to me what practical implicates it has to measure reason about the math abilities with models with a no calculator rule. If someone would build an AGI, it makes sense for the AGI to be able to access subprocesses for tasks like calculators.
An AGI might become a dictator in every country on earth while still not being able to wash dishes or make errors when it comes to driving 100,000 miles. Physical coordination is not required.
How would you expect an AI to take over the world without physical capacity? Attacking financial systems, cybersecurity networks, and computer-operated weapons systems all seem possible from an AI that can simply operate a computer. Is that your vision of an AI takeover, or are there other specific dangerous capabilities you’d like the research community to ensure that AI does not attain?
I’d be genuinely baffled if you think AGI can be imminent at the same time we still don’t have good self-driving cars, robots that can wash dishes, or AI capable of doing well on mathematics word problems. This view would seem to imply that we will get AGI pretty much out of nowhere.
I mean, Eliezer has commented on this position extensively in the AI dialogues. I do think we would likely see AI doing well on mathematics word-problems, but the other two are definitely not things I obviously expect to see before the end (though I do think it’s more likely than not that we would see them).
Zooming out a bit though, I am confused what you are overall responding to with your comment. The thing I am critiquing is not about the “specific capability gains”. It’s just that you are responding to a post saying X, with a bet at odds Y that do not contradict X, and indeed where I think it’s a reasonable common belief to hold both X and Y.
Like, if someone says “it’s ~30% likely” and you say “That seems wrong, I am offering you a bet that you should only take if you have >70% probability on a related hypothesis” then… the obvious response is “but I said I only assign ~30% to this hypothesis, I agree that I assign somewhat more to your weaker hypothesis, but it’s not at all obvious I should assign 70% to it, that’s a big jump”. That’s roughly where I am at.
As the previous OP, to chime in, the specific mechanism by which self-driving cars don’t work but FOOM does is extremely high-capability consequentialist software engineering plus not-much-better-than-today world modeling.
Self-driving and manipulation require incredible-quality video/world modeling, and a bunch of control problems that seem unrelated to symbolic intelligence. Re: solving math problems, that seems way more likely to be a thing such a system could do; the only uncertainty is whether someone invests the time, given it’s not profitable.
You didn’t bet against any of those happening in 5-7 years, though. You bet against it being >25% likely by 2030, or >50% likely in 4. Your bet is completely in concordance with it being more likely than not to happen in 5-7 years.
Do you have any suggestions for what to do when the entire human race is economically obsolete and definitively no longer has any control over its destiny? Your post gives no indication of what you would do or recommend, when your benchmarks are actually surpassed.
Hmm, I guess it’s just really not obvious that your proposed bet here disagrees with the OP. I think I roughly believe both the things that the OP says, and also wouldn’t take your bet. It still feels like a fine bet to offer, but I am confused why it’s phrased so much in contrast to the OP. If you are confident we are not going to see large dangerous capability gains in the next 5-7 years, I think I would much prefer you make a bet that tries to offer corresponding odds and the specific capability gains (though that runs a bit into “betting on doomsday” problems)
What are the “specific cabability gains” you are referring to? I don’t see any specific claims in the post we are responding to. By contrast, we listed 7 concrete tasks that appear trivial to perform if we are AGI-levels of capability, and very easy if we are only a few steps from AGI. I’d be genuinely baffled if you think AGI can be imminent at the same time we still don’t have good self-driving cars, robots that can wash dishes, or AI capable of doing well on mathematics word problems. This view would seem to imply that we will get AGI pretty much out of nowhere.
An AGI might become a dictator in every country on earth while still not being able to wash dishes or make errors when it comes to driving 100,000 miles. Physical coordination is not required.
It’s not clear to me what practical implicates it has to measure reason about the math abilities with models with a no calculator rule. If someone would build an AGI, it makes sense for the AGI to be able to access subprocesses for tasks like calculators.
How would you expect an AI to take over the world without physical capacity? Attacking financial systems, cybersecurity networks, and computer-operated weapons systems all seem possible from an AI that can simply operate a computer. Is that your vision of an AI takeover, or are there other specific dangerous capabilities you’d like the research community to ensure that AI does not attain?
I mean, Eliezer has commented on this position extensively in the AI dialogues. I do think we would likely see AI doing well on mathematics word-problems, but the other two are definitely not things I obviously expect to see before the end (though I do think it’s more likely than not that we would see them).
Zooming out a bit though, I am confused what you are overall responding to with your comment. The thing I am critiquing is not about the “specific capability gains”. It’s just that you are responding to a post saying X, with a bet at odds Y that do not contradict X, and indeed where I think it’s a reasonable common belief to hold both X and Y.
Like, if someone says “it’s ~30% likely” and you say “That seems wrong, I am offering you a bet that you should only take if you have >70% probability on a related hypothesis” then… the obvious response is “but I said I only assign ~30% to this hypothesis, I agree that I assign somewhat more to your weaker hypothesis, but it’s not at all obvious I should assign 70% to it, that’s a big jump”. That’s roughly where I am at.
As the previous OP, to chime in, the specific mechanism by which self-driving cars don’t work but FOOM does is extremely high-capability consequentialist software engineering plus not-much-better-than-today world modeling.
Self-driving and manipulation require incredible-quality video/world modeling, and a bunch of control problems that seem unrelated to symbolic intelligence. Re: solving math problems, that seems way more likely to be a thing such a system could do; the only uncertainty is whether someone invests the time, given it’s not profitable.
You didn’t bet against any of those happening in 5-7 years, though. You bet against it being >25% likely by 2030, or >50% likely in 4. Your bet is completely in concordance with it being more likely than not to happen in 5-7 years.
Do you have any suggestions for what to do when the entire human race is economically obsolete and definitively no longer has any control over its destiny? Your post gives no indication of what you would do or recommend, when your benchmarks are actually surpassed.