I do lean closer to the stance of “whatever we decide based on some ‘reasonable’ reflection process is good”, which seems to encompass a wide range of futures, and seems likely to me to happen by default.
I think I disagree pretty strongly, and this is likely an important crux. Would you be willing to read a couple of articles that point to what I think is convincing contrary evidence? (As you read the first article, consider what would have happened if the people involved had access to AI-enabled commitment or mind-modification technologies.)
http://devinhelton.com/2015/08/dictatorship-and-democracy (I think this is a really good article in its own right but is kind of long, so if you want to skip to the parts most relevant to the current discussion, search for “Tyranny in Maoist China” and “madness spiral”.)
If these articles don’t cause you to update, can you explain why? For example do you think it would be fairly easy to design reflection/deliberation processes that would avoid these pathologies? What about future ones we don’t yet foresee?
… I’m not sure why I used the word “we” in the sentence you quoted. (Maybe I was thinking about a group of value-aligned agents? Maybe I was imagining that “reasonable reflection process” meant that we were in a post-scarcity world, everyone agreed that we should be doing reflection, everyone was already safe? Maybe I didn’t want the definition to sound like I would only care about what I thought and not what everyone else thought? I’m not sure.)
In any case, I think you can change that sentence to “whatever I decide based on some ‘reasonable’ reflection process is good”, and that’s closer to what I meant.
I am much more uncertain about multiagent interactions. Like, suppose we give every person access to a somewhat superintelligent AI assistant that is legitimately trying to help them. Are things okay by default? I lean towards yes, but I’m uncertain. I did read through those two articles, and I broadly buy the theses they advance; I still lean towards yes because:
Things have broadly become better over time, despite the effects that the articles above highlight. The default prediction is that they continue to get better. (And I very uncertainly think people from the past would agree, given enough time to understand our world?)
In general, we learn reasonably well from experience; we try things and they go badly, but then things get better as we learn from that.
Humans tend to be quite risk-averse at trying things, and groups of humans seem to be even more risk-averse. As a result, it seems unlikely that we try a thing that ends up having a “direct” existentially bad effect.
You could worry about an “indirect” existentially bad effect, along the lines of Moloch, where there isn’t any single human’s optimization causing bad things to happen, but selection pressure causes problems. Selection pressure has existed for a long time and hasn’t caused an existentially-bad outcome yet, so the default is that it won’t in the future.
Perhaps AI accelerates the rate of progress in a way where we can’t adapt fast enough, and this is why selection pressures can now cause an existentially bad effect. But this didn’t happen with the Industrial Revolution. (That said, I do find this more plausible than the other scenarios.)
But in fact I usually don’t aim to make claims about these sorts of scenarios; as I mentioned above I’m more optimistic about social solutions (that being the way we have solved this in the past).
In reality, much of the success of a government is due to the role of the particular leaders, particular people, and particular places. If you have a mostly illiterate nation, divided 60%/40% into two tribes, then majoritarian democracy is a really, really bad idea. But if you have a homogeneous, educated, and savvy populace, with a network of private institutions, and a high-trust culture, then many forms of government will work quite well. Much of the purported success of democracy is really survivorship bias. Countries with the most human capital and strongest civic institutions can survive the chaos and demagoguery that comes with regular mass elections. Lesser countries succumb to chaos, and then dictatorship.
I don’t see why having a well-educated populace would protect you against the nigh-inevitable value drift of even well-intentioned leaders when they ascend to power in a highly authoritarian regime.
I agree that if you just had one leader with absolute power then it probably won’t work, and that kind of government probably isn’t included in the author’s “many forms of government will work quite well”. I think what he probably has in mind are governments that look authoritarian from the outside but still has some kind of internal politics/checks-and-balances that can keep the top leader(s) from going off the rails. I wish I had a good gears-level model of how that kind of government/politics works though. I do suspect that “work quite well” might be fragile/temporary and dependent on the top leaders not trying very hard to take absolute power for themselves, but I’m very uncertain about this due to lack of knowledge and expertise.
I think I disagree pretty strongly, and this is likely an important crux. Would you be willing to read a couple of articles that point to what I think is convincing contrary evidence? (As you read the first article, consider what would have happened if the people involved had access to AI-enabled commitment or mind-modification technologies.)
http://devinhelton.com/2015/08/dictatorship-and-democracy (I think this is a really good article in its own right but is kind of long, so if you want to skip to the parts most relevant to the current discussion, search for “Tyranny in Maoist China” and “madness spiral”.)
http://devinhelton.com/historical-amnesia.html
If these articles don’t cause you to update, can you explain why? For example do you think it would be fairly easy to design reflection/deliberation processes that would avoid these pathologies? What about future ones we don’t yet foresee?
… I’m not sure why I used the word “we” in the sentence you quoted. (Maybe I was thinking about a group of value-aligned agents? Maybe I was imagining that “reasonable reflection process” meant that we were in a post-scarcity world, everyone agreed that we should be doing reflection, everyone was already safe? Maybe I didn’t want the definition to sound like I would only care about what I thought and not what everyone else thought? I’m not sure.)
In any case, I think you can change that sentence to “whatever I decide based on some ‘reasonable’ reflection process is good”, and that’s closer to what I meant.
I am much more uncertain about multiagent interactions. Like, suppose we give every person access to a somewhat superintelligent AI assistant that is legitimately trying to help them. Are things okay by default? I lean towards yes, but I’m uncertain. I did read through those two articles, and I broadly buy the theses they advance; I still lean towards yes because:
Things have broadly become better over time, despite the effects that the articles above highlight. The default prediction is that they continue to get better. (And I very uncertainly think people from the past would agree, given enough time to understand our world?)
In general, we learn reasonably well from experience; we try things and they go badly, but then things get better as we learn from that.
Humans tend to be quite risk-averse at trying things, and groups of humans seem to be even more risk-averse. As a result, it seems unlikely that we try a thing that ends up having a “direct” existentially bad effect.
You could worry about an “indirect” existentially bad effect, along the lines of Moloch, where there isn’t any single human’s optimization causing bad things to happen, but selection pressure causes problems. Selection pressure has existed for a long time and hasn’t caused an existentially-bad outcome yet, so the default is that it won’t in the future.
Perhaps AI accelerates the rate of progress in a way where we can’t adapt fast enough, and this is why selection pressures can now cause an existentially bad effect. But this didn’t happen with the Industrial Revolution. (That said, I do find this more plausible than the other scenarios.)
But in fact I usually don’t aim to make claims about these sorts of scenarios; as I mentioned above I’m more optimistic about social solutions (that being the way we have solved this in the past).
Not on topic, but from the first article:
I don’t see why having a well-educated populace would protect you against the nigh-inevitable value drift of even well-intentioned leaders when they ascend to power in a highly authoritarian regime.
I agree that if you just had one leader with absolute power then it probably won’t work, and that kind of government probably isn’t included in the author’s “many forms of government will work quite well”. I think what he probably has in mind are governments that look authoritarian from the outside but still has some kind of internal politics/checks-and-balances that can keep the top leader(s) from going off the rails. I wish I had a good gears-level model of how that kind of government/politics works though. I do suspect that “work quite well” might be fragile/temporary and dependent on the top leaders not trying very hard to take absolute power for themselves, but I’m very uncertain about this due to lack of knowledge and expertise.