Whereas this post seems to suggest the response of: Oh well, I guess it’s a dice roll regardless of what sort of AI we build. Which is giving up awfully quickly, as if we had exhausted the design space for possible AIs and seen that there was no way to move forward with a large chance at a big flourishing future.
I dispute that I’m “giving up” in any meaningful sense here. I’m happy to consider alternative proposals for how we could make the future large and flourishing from a total utilitarian perspective rather than merely trying to solve technical alignment problems. The post itself was simply intended to discuss the moral implications of AI alignment (itself a massive topic), but it was not intended to be an exhaustive survey of everything we can do to make the future go better. I agree we should aim high, in any case.
This response also doesn’t seem very quantitative—it goes very quickly from the idea that an aligned AI might not get a big flourishing future, to the view that alignment is “neutral” as if the chances of getting a big flourishing future were identically small under both options. But the obvious question for a total utilitarian who does wind up with just 2 options, each of which is a dice roll, is Which set of dice has better odds?
I don’t think this choice is literally a coin flip in expected value, and I agree that one might lean in one direction over the other. However, I think it’s quite hard to quantify this question meaningfully. My personal conclusion is simply that Iam not swayed in any particular direction on this question; I am currently suspending judgement. I think one could reasonably still think that it’s more like 60-40 thing than a 40-60 thing or 50-50 coin flip. But I guess in this case, I wanted to let my readers decide for themselves which of these numbers they want to take away from what I wrote, rather than trying to pin down a specific number for them.
I dispute that I’m “giving up” in any meaningful sense here. I’m happy to consider alternative proposals for how we could make the future large and flourishing from a total utilitarian perspective rather than merely trying to solve technical alignment problems. The post itself was simply intended to discuss the moral implications of AI alignment (itself a massive topic), but it was not intended to be an exhaustive survey of everything we can do to make the future go better. I agree we should aim high, in any case.
I don’t think this choice is literally a coin flip in expected value, and I agree that one might lean in one direction over the other. However, I think it’s quite hard to quantify this question meaningfully. My personal conclusion is simply that I am not swayed in any particular direction on this question; I am currently suspending judgement. I think one could reasonably still think that it’s more like 60-40 thing than a 40-60 thing or 50-50 coin flip. But I guess in this case, I wanted to let my readers decide for themselves which of these numbers they want to take away from what I wrote, rather than trying to pin down a specific number for them.