Let’s say you currently think the singularity will proceed at a rate of R.
What does this mean? On my understanding, singularities don’t proceed at fixed rates?
I agree that in practice there will be some maximum rate of GDP growth, because there are fundamental physical limits (and more tight in-practice limits that we don’t know), but it seems like they’ll be way higher than 25% per year. Or to put it a different way, at 25% max rate I think it stops deserving the term “singularity”, it seems like it takes decades and maybe centuries to reach technological maturity at that rate. (Which could totally happen! Maybe we will move very slowly and cautiously! I don’t particularly expect it though.)
If you actually mean halving the peak rate of GDP growth during the singularity, and a singularity actually happens, then I think it doesn’t affect my actions at all; all of the relevant stuff happened well before we get to the peak rate.
If you ask me to imagine “max rates at orders of magnitude where Rohin would say there was no singularity”, then I think I pivot my plan for impact into figuring out how exactly we are going to manage to coordinate to move slowly even when there’s tons of obvious value lying around, and then trying to use the same techniques to get tons and tons of oversight on the systems we build.
You learn that the cost of misalignment is half as much as you thought, in the sense that slightly misaligned AI software impose costs that are half as much (ethically, or economically), compared to what you used to think.
Hmm, I interpreted “cost of misalignment” as “expected cost of misalignment”, as the standard way to deal with probabilistic things, but it sounds like you want something else.
Let’s say for purposes of argument I think 10% chance of extinction, and 90% chance of “moderate costs but nothing terrible”. Which of the following am I supposed to have updated to?
5% extinction, 95% moderate costs
5% extinction, 45% moderate costs, 50% perfect world
I was imagining (4), but any of (1) - (3) would also not change my actions; it sounds like you want me to imagine (5) but in that case I just completely switch out of AI alignment and work on something else, but that’s because you moved p(extinction) from 10% to 0%, which is a wild update to have made and not what I would call “the cost of misalignment is half as much as I thought”.
I think you sufficiently addressed my confusion, so you don’t need to reply to this comment, but I still had a few responses to what you said.
What does this mean? On my understanding, singularities don’t proceed at fixed rates?
No, I agree. But growth is generally measured over an interval. In the original comment I proposed the interval of one year during the peak rate of economic growth. To allay your concern that a 25% growth rate indicates we didn’t experience a singularity, I meant that we were halving the growth rate during the peak economic growth year in our future, regardless of whether that rate was very fast.
I agree that in practice there will be some maximum rate of GDP growth, because there are fundamental physical limits (and more tight in-practice limits that we don’t know), but it seems like they’ll be way higher than 25% per year.
The 25% figure was totally arbitrary. I didn’t mean it as any sort of prediction. I agree that an extrapolation from biological growth implies that we can and should see >1000% growth rates eventually, though it seems plausible that we would coordinate to avoid that.
If you actually mean halving the peak rate of GDP growth during the singularity, and a singularity actually happens, then I think it doesn’t affect my actions at all; all of the relevant stuff happened well before we get to the peak rate.
That’s reasonable. A separate question might be about whether the rate of growth during the entire duration from now until the peak rate will cut in half.
Let’s say for purposes of argument I think 10% chance of extinction, and 90% chance of “moderate costs but nothing terrible”. Which of the following am I supposed to have updated to?
I think the way you’re bucketing this into “costs if we go extinct” and “costs if we don’t go extinct” is reasonable. But one could also think that the disvalue of extinction is more continuous with disvalue in non-extinction scenarios, which makes things a bit more tricky. I hope that makes sense.
But one could also think that the disvalue of extinction is more continuous with disvalue in non-extinction scenarios, which makes things a bit more tricky.
I’m happy to use continuous notions (and that’s what I was doing in my original comment) as long as “half the cost” means “you update such that the expected costs of misalignment according to your probability distribution over the future are halved”. One simple way to imagine this update is to take all the worlds where there was any misalignment, halve their probability, and distribute the extra probability mass to worlds with zero costs of misalignment. At which point I reason “well, 10% extinction changes to 5% extinction, I don’t need to know anything else to know that I’m still going to work on alignment, and given that, none of my actions are going to change (since the relative probabilities of different misalignment failure scenarios remain the same, which is what determines my actions within alignment)”.
I got the sense from your previous comment that you wanted me to imagine some different form of update and I was trying to figure out what.
What does this mean? On my understanding, singularities don’t proceed at fixed rates?
I agree that in practice there will be some maximum rate of GDP growth, because there are fundamental physical limits (and more tight in-practice limits that we don’t know), but it seems like they’ll be way higher than 25% per year. Or to put it a different way, at 25% max rate I think it stops deserving the term “singularity”, it seems like it takes decades and maybe centuries to reach technological maturity at that rate. (Which could totally happen! Maybe we will move very slowly and cautiously! I don’t particularly expect it though.)
If you actually mean halving the peak rate of GDP growth during the singularity, and a singularity actually happens, then I think it doesn’t affect my actions at all; all of the relevant stuff happened well before we get to the peak rate.
If you ask me to imagine “max rates at orders of magnitude where Rohin would say there was no singularity”, then I think I pivot my plan for impact into figuring out how exactly we are going to manage to coordinate to move slowly even when there’s tons of obvious value lying around, and then trying to use the same techniques to get tons and tons of oversight on the systems we build.
Hmm, I interpreted “cost of misalignment” as “expected cost of misalignment”, as the standard way to deal with probabilistic things, but it sounds like you want something else.
Let’s say for purposes of argument I think 10% chance of extinction, and 90% chance of “moderate costs but nothing terrible”. Which of the following am I supposed to have updated to?
5% extinction, 95% moderate costs
5% extinction, 45% moderate costs, 50% perfect world
10% extinction, 90% mild costs
10% outcome-half-as-bad-as-extinction, 90% mild costs
0% extinction, 100% mild costs
I was imagining (4), but any of (1) - (3) would also not change my actions; it sounds like you want me to imagine (5) but in that case I just completely switch out of AI alignment and work on something else, but that’s because you moved p(extinction) from 10% to 0%, which is a wild update to have made and not what I would call “the cost of misalignment is half as much as I thought”.
I think you sufficiently addressed my confusion, so you don’t need to reply to this comment, but I still had a few responses to what you said.
No, I agree. But growth is generally measured over an interval. In the original comment I proposed the interval of one year during the peak rate of economic growth. To allay your concern that a 25% growth rate indicates we didn’t experience a singularity, I meant that we were halving the growth rate during the peak economic growth year in our future, regardless of whether that rate was very fast.
The 25% figure was totally arbitrary. I didn’t mean it as any sort of prediction. I agree that an extrapolation from biological growth implies that we can and should see >1000% growth rates eventually, though it seems plausible that we would coordinate to avoid that.
That’s reasonable. A separate question might be about whether the rate of growth during the entire duration from now until the peak rate will cut in half.
I think the way you’re bucketing this into “costs if we go extinct” and “costs if we don’t go extinct” is reasonable. But one could also think that the disvalue of extinction is more continuous with disvalue in non-extinction scenarios, which makes things a bit more tricky. I hope that makes sense.
Cool, that all makes sense.
I’m happy to use continuous notions (and that’s what I was doing in my original comment) as long as “half the cost” means “you update such that the expected costs of misalignment according to your probability distribution over the future are halved”. One simple way to imagine this update is to take all the worlds where there was any misalignment, halve their probability, and distribute the extra probability mass to worlds with zero costs of misalignment. At which point I reason “well, 10% extinction changes to 5% extinction, I don’t need to know anything else to know that I’m still going to work on alignment, and given that, none of my actions are going to change (since the relative probabilities of different misalignment failure scenarios remain the same, which is what determines my actions within alignment)”.
I got the sense from your previous comment that you wanted me to imagine some different form of update and I was trying to figure out what.