I might disagree with you epistemically but… what do I have to win if AGI happens before 2030 and I win the bet? I don’t think either of us will still care about our bet after that happens. Doesn’t this just run into all the standard problems of predicting doomsday?
Edit: Oh, I also just saw you meant 3:1 odds in your favor. That’s… weird, since it doesn’t even disagree with the OP? Why would the OP take the bet that you propose, given they only assign ~30% probability to this outcome?
Bryan Caplan and Eliezer are resolving their Doomsday bet by having Bryan Caplan pay Eliezer upfront and if the doomsday scenario does not happen by Jan 1 2030, Eliezer will give Bryan his payout. It’s a pretty method for betting on doomsday.
Why would the OP take the bet that you propose, given they only assign ~30% probability to this outcome?
The conditions we offered fall well short of AGI, so it seems reasonable that the author would assign way more than 30% to this outcome. Furthermore, we offered a 1:1 bet for January 1st 2026.
Edit: The OP also says, “Crying wolf isn’t really a thing here; the societal impact of these capabilities is undeniable and you will not lose credibility even if 3 years from now these systems haven’t yet FOOMed, because the big changes will be obvious and you’ll have predicted that right.” which seems to imply that we will likely obtain very impressive capabilities within 3 years. In my opinion, this statement is directly challenged by our 1:1 bet.
Hmm, I guess it’s just really not obvious that your proposed bet here disagrees with the OP. I think I roughly believe both the things that the OP says, and also wouldn’t take your bet. It still feels like a fine bet to offer, but I am confused why it’s phrased so much in contrast to the OP. If you are confident we are not going to see large dangerous capability gains in the next 5-7 years, I think I would much prefer you make a bet that tries to offer corresponding odds and the specific capability gains (though that runs a bit into “betting on doomsday” problems)
If you are confident we are not going to see large dangerous capability gains in the next 5-7 years, I think I would much prefer you make a bet that tries to offer corresponding odds and the specific capability gains (though that runs a bit into “betting on doomsday” problems)
What are the “specific cabability gains” you are referring to? I don’t see any specific claims in the post we are responding to. By contrast, we listed 7 concrete tasks that appear trivial to perform if we are AGI-levels of capability, and very easy if we are only a few steps from AGI. I’d be genuinely baffled if you think AGI can be imminent at the same time we still don’t have good self-driving cars, robots that can wash dishes, or AI capable of doing well on mathematics word problems. This view would seem to imply that we will get AGI pretty much out of nowhere.
An AGI might become a dictator in every country on earth while still not being able to wash dishes or make errors when it comes to driving 100,000 miles. Physical coordination is not required.
It’s not clear to me what practical implicates it has to measure reason about the math abilities with models with a no calculator rule. If someone would build an AGI, it makes sense for the AGI to be able to access subprocesses for tasks like calculators.
An AGI might become a dictator in every country on earth while still not being able to wash dishes or make errors when it comes to driving 100,000 miles. Physical coordination is not required.
How would you expect an AI to take over the world without physical capacity? Attacking financial systems, cybersecurity networks, and computer-operated weapons systems all seem possible from an AI that can simply operate a computer. Is that your vision of an AI takeover, or are there other specific dangerous capabilities you’d like the research community to ensure that AI does not attain?
I’d be genuinely baffled if you think AGI can be imminent at the same time we still don’t have good self-driving cars, robots that can wash dishes, or AI capable of doing well on mathematics word problems. This view would seem to imply that we will get AGI pretty much out of nowhere.
I mean, Eliezer has commented on this position extensively in the AI dialogues. I do think we would likely see AI doing well on mathematics word-problems, but the other two are definitely not things I obviously expect to see before the end (though I do think it’s more likely than not that we would see them).
Zooming out a bit though, I am confused what you are overall responding to with your comment. The thing I am critiquing is not about the “specific capability gains”. It’s just that you are responding to a post saying X, with a bet at odds Y that do not contradict X, and indeed where I think it’s a reasonable common belief to hold both X and Y.
Like, if someone says “it’s ~30% likely” and you say “That seems wrong, I am offering you a bet that you should only take if you have >70% probability on a related hypothesis” then… the obvious response is “but I said I only assign ~30% to this hypothesis, I agree that I assign somewhat more to your weaker hypothesis, but it’s not at all obvious I should assign 70% to it, that’s a big jump”. That’s roughly where I am at.
As the previous OP, to chime in, the specific mechanism by which self-driving cars don’t work but FOOM does is extremely high-capability consequentialist software engineering plus not-much-better-than-today world modeling.
Self-driving and manipulation require incredible-quality video/world modeling, and a bunch of control problems that seem unrelated to symbolic intelligence. Re: solving math problems, that seems way more likely to be a thing such a system could do; the only uncertainty is whether someone invests the time, given it’s not profitable.
You didn’t bet against any of those happening in 5-7 years, though. You bet against it being >25% likely by 2030, or >50% likely in 4. Your bet is completely in concordance with it being more likely than not to happen in 5-7 years.
Do you have any suggestions for what to do when the entire human race is economically obsolete and definitively no longer has any control over its destiny? Your post gives no indication of what you would do or recommend, when your benchmarks are actually surpassed.
If you expect the apocalypse to happen by a given date, you should rationally value having money then much less than the market(if the market doesn’t expect the apocalypse). So you can simulate a bet by having an apocalypse-expecter take a high-interest-rate loan from an apocalypse-denier, paying the loan back(if the world survives) at the date of the purported apocalypse(h/t daniel filan).
Couldn’t they just get lower interest rate loans elsewhere?
Or, interest doesn’t start until the bet outcome date passes? I’d give the apocalypse-expecter $1000 now, and they pay me back with interest when the outcome date passes, with no interest payments before then.
For those wanting to lend out money to gain interest on and use that money for EA causes, this might be useful:
Ya, I was thinking this, too, but they could possibly get a lot of loans or much larger loans at lower interest rates, and it’s not clear when they would start looking at this one as the next best to pursue. Maybe it’s more time-efficient (more loaned money per hour spent setting up and dealing with) to take this kind of AI-bet loan, though, but $1000 is very low.
We also propose betting using a mechanism that mitigates some of these issues:
Since we recognize that betting incentives can be weak over long time-horizons, we are also offering the option of employing Tamay’s recently described betting procedure in which we would enter a series of repeated 2-year contracts until the resolution date.
I might disagree with you epistemically but… what do I have to win if AGI happens before 2030 and I win the bet? I don’t think either of us will still care about our bet after that happens. Doesn’t this just run into all the standard problems of predicting doomsday?
Edit: Oh, I also just saw you meant 3:1 odds in your favor. That’s… weird, since it doesn’t even disagree with the OP? Why would the OP take the bet that you propose, given they only assign ~30% probability to this outcome?
Bryan Caplan and Eliezer are resolving their Doomsday bet by having Bryan Caplan pay Eliezer upfront and if the doomsday scenario does not happen by Jan 1 2030, Eliezer will give Bryan his payout. It’s a pretty method for betting on doomsday.
The conditions we offered fall well short of AGI, so it seems reasonable that the author would assign way more than 30% to this outcome. Furthermore, we offered a 1:1 bet for January 1st 2026.
Edit: The OP also says, “Crying wolf isn’t really a thing here; the societal impact of these capabilities is undeniable and you will not lose credibility even if 3 years from now these systems haven’t yet FOOMed, because the big changes will be obvious and you’ll have predicted that right.” which seems to imply that we will likely obtain very impressive capabilities within 3 years. In my opinion, this statement is directly challenged by our 1:1 bet.
Hmm, I guess it’s just really not obvious that your proposed bet here disagrees with the OP. I think I roughly believe both the things that the OP says, and also wouldn’t take your bet. It still feels like a fine bet to offer, but I am confused why it’s phrased so much in contrast to the OP. If you are confident we are not going to see large dangerous capability gains in the next 5-7 years, I think I would much prefer you make a bet that tries to offer corresponding odds and the specific capability gains (though that runs a bit into “betting on doomsday” problems)
What are the “specific cabability gains” you are referring to? I don’t see any specific claims in the post we are responding to. By contrast, we listed 7 concrete tasks that appear trivial to perform if we are AGI-levels of capability, and very easy if we are only a few steps from AGI. I’d be genuinely baffled if you think AGI can be imminent at the same time we still don’t have good self-driving cars, robots that can wash dishes, or AI capable of doing well on mathematics word problems. This view would seem to imply that we will get AGI pretty much out of nowhere.
An AGI might become a dictator in every country on earth while still not being able to wash dishes or make errors when it comes to driving 100,000 miles. Physical coordination is not required.
It’s not clear to me what practical implicates it has to measure reason about the math abilities with models with a no calculator rule. If someone would build an AGI, it makes sense for the AGI to be able to access subprocesses for tasks like calculators.
How would you expect an AI to take over the world without physical capacity? Attacking financial systems, cybersecurity networks, and computer-operated weapons systems all seem possible from an AI that can simply operate a computer. Is that your vision of an AI takeover, or are there other specific dangerous capabilities you’d like the research community to ensure that AI does not attain?
I mean, Eliezer has commented on this position extensively in the AI dialogues. I do think we would likely see AI doing well on mathematics word-problems, but the other two are definitely not things I obviously expect to see before the end (though I do think it’s more likely than not that we would see them).
Zooming out a bit though, I am confused what you are overall responding to with your comment. The thing I am critiquing is not about the “specific capability gains”. It’s just that you are responding to a post saying X, with a bet at odds Y that do not contradict X, and indeed where I think it’s a reasonable common belief to hold both X and Y.
Like, if someone says “it’s ~30% likely” and you say “That seems wrong, I am offering you a bet that you should only take if you have >70% probability on a related hypothesis” then… the obvious response is “but I said I only assign ~30% to this hypothesis, I agree that I assign somewhat more to your weaker hypothesis, but it’s not at all obvious I should assign 70% to it, that’s a big jump”. That’s roughly where I am at.
As the previous OP, to chime in, the specific mechanism by which self-driving cars don’t work but FOOM does is extremely high-capability consequentialist software engineering plus not-much-better-than-today world modeling.
Self-driving and manipulation require incredible-quality video/world modeling, and a bunch of control problems that seem unrelated to symbolic intelligence. Re: solving math problems, that seems way more likely to be a thing such a system could do; the only uncertainty is whether someone invests the time, given it’s not profitable.
You didn’t bet against any of those happening in 5-7 years, though. You bet against it being >25% likely by 2030, or >50% likely in 4. Your bet is completely in concordance with it being more likely than not to happen in 5-7 years.
Do you have any suggestions for what to do when the entire human race is economically obsolete and definitively no longer has any control over its destiny? Your post gives no indication of what you would do or recommend, when your benchmarks are actually surpassed.
Just for the record, I regret that statement, independent of making a bet or not.
If you expect the apocalypse to happen by a given date, you should rationally value having money then much less than the market(if the market doesn’t expect the apocalypse). So you can simulate a bet by having an apocalypse-expecter take a high-interest-rate loan from an apocalypse-denier, paying the loan back(if the world survives) at the date of the purported apocalypse(h/t daniel filan).
Couldn’t they just get lower interest rate loans elsewhere?
Or, interest doesn’t start until the bet outcome date passes? I’d give the apocalypse-expecter $1000 now, and they pay me back with interest when the outcome date passes, with no interest payments before then.
For those wanting to lend out money to gain interest on and use that money for EA causes, this might be useful:
https://founderspledge.com/stories/investing-to-give
This doesn’t mean necessarily that you shouldn’t take the bet, but maybe that you should also take the loan.
Ya, I was thinking this, too, but they could possibly get a lot of loans or much larger loans at lower interest rates, and it’s not clear when they would start looking at this one as the next best to pursue. Maybe it’s more time-efficient (more loaned money per hour spent setting up and dealing with) to take this kind of AI-bet loan, though, but $1000 is very low.
Yeah, this is what I had in mind. There wouldn’t be interest payments until the date of the apocalypse.
We also propose betting using a mechanism that mitigates some of these issues:
Also, we give odds of 1:1 if anyone wants to take us up on the bet for January 1st 2026.