I think this dialogue would have benefitted from some more specifics in two areas:
Some specific object level disagreements with respect to “but it doesn’t seem to me to justify the second clause ‘implying that these instrumental values are likely to be pursued by many intelligent agents.’” would have been helpful. For example Luke could claim that “get lots of computational power” or “understand physics” is something of a convergent instrumental goal and Ben could say why he doesn’t think that’s true.
“Calculating the decimals of pi may be a fairly simple mathematical operation that doesn’t have any need for superintelligence, and thus may be a really unlikely goal for a superintelligence—so that if you tried to build a superintelligence with this goal and connected it to the real world, it would very likely get its initial goal subverted and wind up pursuing some different, less idiotic goal.” I think I would have understood this better with some specific examples of how the initial goal might be subverted. For example “AI researcher makes an AI to calculate decimals of PI as an experiment, but when it starts getting more powerful, he decides that’s a stupid goal and gives it something more reasonable”
Some specific object level disagreements with respect to “but it doesn’t seem to me to justify the second clause ‘implying that these instrumental values are likely to be pursued by many intelligent agents.’” would have been helpful. For example Luke could claim that “get lots of computational power” or “understand physics” is something of a convergent instrumental goal and Ben could say why he doesn’t think that’s true.
He could—if that was his position. However, AFAICS, that’s not what the debate is about. Everyone agrees that those are convergent instrumental goals—the issue is more whether machinines that we build are likely to follow them to the detriment of the surrounding humans—or be programmed to behave otherwise.
I think this dialogue would have benefitted from some more specifics in two areas:
Some specific object level disagreements with respect to “but it doesn’t seem to me to justify the second clause ‘implying that these instrumental values are likely to be pursued by many intelligent agents.’” would have been helpful. For example Luke could claim that “get lots of computational power” or “understand physics” is something of a convergent instrumental goal and Ben could say why he doesn’t think that’s true.
“Calculating the decimals of pi may be a fairly simple mathematical operation that doesn’t have any need for superintelligence, and thus may be a really unlikely goal for a superintelligence—so that if you tried to build a superintelligence with this goal and connected it to the real world, it would very likely get its initial goal subverted and wind up pursuing some different, less idiotic goal.” I think I would have understood this better with some specific examples of how the initial goal might be subverted. For example “AI researcher makes an AI to calculate decimals of PI as an experiment, but when it starts getting more powerful, he decides that’s a stupid goal and gives it something more reasonable”
He could—if that was his position. However, AFAICS, that’s not what the debate is about. Everyone agrees that those are convergent instrumental goals—the issue is more whether machinines that we build are likely to follow them to the detriment of the surrounding humans—or be programmed to behave otherwise.
I see, that wasn’t very clear to me. I think giving some specific examples which exemplify the disagreement would have helped clarify that for me.