You probably already agreed with “Ghosts in the Machine” before reading it since obviously, a program executes exactly its code even in the context of AI. Also obviously, the program can still appear to not do what it’s supposed to if “supposed” is taken to mean to programmer’s intent.
These statements don’t ignore machine learning; they imply that we should not try to build an FAI using current machine learning techniques. You’re right, we understand (program + parameters learned from dataset) even less than (program). So while the outside view might say: “current machine learning techniques are very powerful, so they are likely to be used for FAI,” that piece of inside view says: “actually, they aren’t. Or at least they shouldn’t.” (“learn” has a precise operational meaning here, so this is unrelated to whether an FAI should “learn” in some other sense of the word).
Again, whether a development has been successful or promising in some field doesn’t mean it will be as successful in FAI, so imitation of the human brain isn’t necessarily good here. Reasoning by analogy and thinking about evolution is also unlikely to help; nature may have given us “goals”, but they are not goals in the same sense as : “The goal of this function is to add 2 to its input,” or “The goal of this program is to play chess well,” or “The goal of this FAI is to maximize human utility.”
I think practitioners of ML should be more wary of their tools. I’m not saying ML is a fast track to strong AI, just that we don’t know if it is. Several ML people voiced reassurances recently, but I would have expected them to do that even if it was possible to detect danger at this point. So I think someone should find a way to make the field more careful.
I don’t think that someone should be MIRI though; status differences are too high, they are not insiders, etc. My best bet would be a prominent ML researcher starting to speak up and giving detailed, plausible hypotheticals in public (I mean near-future hypotheticals where some error creates a lot of trouble for everyone).
You probably already agreed with “Ghosts in the Machine” before reading it since obviously, a program executes exactly its code even in the context of AI. Also obviously, the program can still appear to not do what it’s supposed to if “supposed” is taken to mean to programmer’s intent.
These statements don’t ignore machine learning; they imply that we should not try to build an FAI using current machine learning techniques. You’re right, we understand (program + parameters learned from dataset) even less than (program). So while the outside view might say: “current machine learning techniques are very powerful, so they are likely to be used for FAI,” that piece of inside view says: “actually, they aren’t. Or at least they shouldn’t.” (“learn” has a precise operational meaning here, so this is unrelated to whether an FAI should “learn” in some other sense of the word).
Again, whether a development has been successful or promising in some field doesn’t mean it will be as successful in FAI, so imitation of the human brain isn’t necessarily good here. Reasoning by analogy and thinking about evolution is also unlikely to help; nature may have given us “goals”, but they are not goals in the same sense as : “The goal of this function is to add 2 to its input,” or “The goal of this program is to play chess well,” or “The goal of this FAI is to maximize human utility.”
Buit people are using ML techniques. Should MIRI be campaigning to get this research stopped?
I think practitioners of ML should be more wary of their tools. I’m not saying ML is a fast track to strong AI, just that we don’t know if it is. Several ML people voiced reassurances recently, but I would have expected them to do that even if it was possible to detect danger at this point. So I think someone should find a way to make the field more careful.
I don’t think that someone should be MIRI though; status differences are too high, they are not insiders, etc. My best bet would be a prominent ML researcher starting to speak up and giving detailed, plausible hypotheticals in public (I mean near-future hypotheticals where some error creates a lot of trouble for everyone).