First, thank you for publishing this illuminating exchange.
I must say that Pei Wang sounds way more convincing to an uninitiated, but curious and mildly intelligent lay person (that would be me). Does not mean he is right, but he sure does make sense.
When Luke goes on to make a point, I often get lost in a jargon (“manifest convergent instrumental goals”) or have to look up a paper that Pei (or other AGI researchers) does not hold in high regard. When Pei Wang makes an argument, it is intuitively clear and does not require going through a complex chain of reasoning outlined in the works of one Eliezer Yudkowsky and not vetted by the AI community at large. This is, of course, not a guarantee of its validity, but it sure is easier to follow.
Some of the statements are quite damning, actually: “The “friendly AI” approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most AGI researchers. The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.” If one were to replace AI with physics, I would tend to dismiss EY as a crank just based on this statement, assuming it is accurate.
What makes me trust Pei Wang more than Luke is the common-sense statements like “to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety.” and “unless you get a right idea about what AGI is and how it can be built, it is very unlikely for you to know how to make it safe”. Similarly, the SIAI position of “accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI first, rather than arbitrary superhuman AGI” rubs me the wrong way. While it does not necessarily mean it is wrong, the inability to convince outside experts that it is right is not a good sign.
This might be my confirmation bias, but I would be hard pressed to disagree with “To develop a non-trivial education theory of AGI requires a good understanding about how the system works, so if we don’t know how to build an AGI, there is no chance for us to know how to make it safe. I don’t think a good education theory can be “proved” in advance, pure theoretically. Rather, we’ll learn most of it by interacting with baby AGIs, just like how many of us learn how to educate children.”
As a side point, I cannot help but wonder if the outcome of this discussion would have been different were it EY and not LM involved in it.
What makes me trust Pei Wang more than Luke is the common-sense statements like “to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety.”...
This sort of “common sense” can be highly misleading! For example, here Wang is drawing parallels between a nascent AI and a human child to argue about nature vs nurture. But if we compare a human and a different social animal, we’ll see that most of the differences in their behavior are innate and the gap can’t be covered by any amount of “education”: e.g. humans can’t really become as altruistic and self-sacrificing as worker ants because they’ll still retain some self-preservation instinct, no matter how you brainwash them.
What makes Wang think that this sort of fixed attitude—which can be made more hard-wired than the instincts of biological organisms—cannot manifest itself in an AGI?
(I’m certain that a serious AI thinker, or just someone with good logic and clear thinking, could find a lot more holes in such “common sense” talk.)
What makes Wang think that this sort of fixed attitude—which can be made more hard-wired than the instincts of biological organisms—cannot manifest itself in an AGI?
Presumably the argument is something like:
You can’t build an AI that is intelligent from the moment you switch it on: you have to train it.
We know how to train intelligence into humans, its called education
An AI that lacked human-style instincts and learning abilities at switch-on wouldn’t be trainable by us, we just wouldn’t know how, so it would never reach intelligence.
When Pei Wang makes an argument, it is intuitively clear and does not require going through a complex chain of reasoning […]
I felt the main reason was anthropomorphism:
Such a computer system will share many properties with the human mind; […]
If intelligence turns out to be adaptive (as believed by me and many others), then a “friendly AI” will be mainly the result of proper education, not proper design.
Note that I don’t want to accuse Pei Wang of anthropomorphism. My point is, his choice of words appeal to our anthropomorphism, which is highly intuitive. Another example of an highly intuitive, but not very helpful sentence:
It is my belief that an AGI will necessarily be adaptive, which implies that the goals it actively pursues constantly change as a function of its experience, and are not fully restricted by its initial (given) goals.
Intuitive, because applied to humans, we can easily see that we can change plans according to experience. Like apply for a PhD, then dropping out when finding out you don’t enjoy it after all. You can abandon the goal of making research, and have a new goal of, say, practicing and teaching surfing.
Not very helpful, because the split between initial goals and later goals does not help you build an AI that will actually do something “good”. Here, the split between instrumental goals (means to an end), and terminal goals (the AI’s “ulterior motives”) is more important. To give a human example, in the case above, doing research or surfing are both means to the same end (like being happy, or something more careful but so complicated nobody knows how to clearly specify it yet). For an AI, as Pei Wang implies, the initial goals aren’t necessarily supposed to constraint all future goals. But its terminal goals are indeed supposed to constrain the instrumental goals it will form later. (More precisely, the instrumental goals are supposed to follow from the terminal goals and the AI’s current model of the world.)
Edit: it just occurred to me, that terminal goals have somehow to be encoded into the AI before we set it loose. They are necessarily initial goals (if they aren’t, the AI is by definition unfriendly —not a problem if its goals miraculously converge towards something “good”, though). Thinking about it, it looks like Pei Wang doesn’t believe it is possible to make an AI with stable terminal goals.
I think I am in the same position as you are (uninitiated but curious) and I had the same immediate reaction that Pei was more convincing. However, for me, I think this was the result of two factors
Pei is a Professor
Pei treated the interview like a conversation with someone who has read a couple books and that’s about it.
Maybe the 2nd point isn’t entirely true, but that was what immediately stuck out after thinking about why I was drawn to Pei’s arguments. Once I eliminated his status as a barometer for his arguments… it just became (1) an issue of my own lack of knowledge and (2) the tone of the responses.
For one thing, why the hell should I understand this in the first place? This is a dialogue between two prominent AI researchers. What I would expect from such a dialogue would be exactly what I would expect from sitting in on a graduate philosophy seminar or a computer science colloquium—I would be able to follow the gist of it, but not the gritty details. I would expect to hear some complex arguments that would require a couple textbooks and a dozen tabs open in my browser to be able to follow.
But I was able to understand Pei’s arguments and play with them! If solving these kinds of conceptual problems is this easy, I might try to take over the world myself.
Not to say that the appearance of “complexity” is necessary for a good argument (EY’s essays are proof), but here it seems like this lack of complexity (or as someone else said, the appeal to common sense) is a warning for the easily persuaded. Rereading with these things in mind illuminates the discussion a bit better.
I was actually a bit depressed by this dialogue. It seemed like an earnest (but maybe a little over the top with the LW references) attempt by lukeprog to communicate interesting ideas. I may be setting my expectations a little high, but Pei seemed to think he was engaging an undergraduate asking about sorting algorithms.
Of course, I could be completely misinterpreting things. I thought I would share my thought process after I came to the same conclusion as you did.
I may be setting my expectations a little high, but Pei seemed to think he was engaging an undergraduate asking about sorting algorithms.
And he thought the undergrad terribly naive for not understanding that all sorting algorithms are actually just bubble sort.
This is why I find that unless the individual is remarkably open—to the point of being peculiar—it is usually pointless to try to communicate across status barriers. Status makes people (have the tendency and social incentives that make them act) stupid when it comes to comprehending others.
This is a dialogue between two prominent AI researchers.
What? This was a dialog between Pei and lukeprog, right?
Of course, I could be completely misinterpreting things. I thought I would share my thought process after I came to the same conclusion as you did.
I’m curious about what you mean by the appellation “prominent AI researcher” that you would apply it to lukeprog, and whether he considers himself as a member of that category.
What makes me trust Pei Wang more than Luke is the common-sense statements like “to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety.” and “unless you get a right idea about what AGI is and how it can be built, it is very unlikely for you to know how to make it safe”.
Um… but these are statements I agreed with.
I wish Pei had taken the time to read the articles I repeatedly linked to, for they were written precisely to explain why his position is misguided.
I wish Pei had taken the time to read the articles I repeatedly linked to, for they were written precisely to explain why his position is misguided.
I think you should have listed a couple of the most important articles at the beginning as necessary background reading to understand your positions and terminology (like Pei did with his papers), and then only used links very sparingly afterwards. Unless you already know your conversation partner takes you very seriously, you can’t put 5 hyperlinks in an email and expect the other person to read them all. When they see that many links, they’ll probably just ignore all of them. (Not to mention the signaling issues that others already pointed out.)
I wish Pei had taken the time to read the articles I repeatedly linked to, for they were written precisely to explain why his position is misguided.
The reactions I got (from a cognitive scientist and another researcher) is that Bostrom is a “sloppy thinker” (original words) and that SI’s understanding of AGI is naive.
Michael Littman told me he is going to read some of the stuff too. I haven’t got an answer yet though.
Hmm, maybe it is possible to summarize them in a language that an AI expert would find both meaningful and convincing. How is your mental model of Dr Wang?
The title of Professor supersedes the title of Doctor, at least in the case of a PhD (I’m not sure about MD, but would assume similarly). His CV indicates pretty clearly that he is an Associate Professor at temple university, so the correct title is Professor.
Again, I am being somewhat super-pedantic here, and I apologize for any annoyance this causes. But hopefully it will help you in your future signalling endeavors.
Also, in most situations it is okay to just go by first name, or full name (without any titles); I have I think exclusively referred to Pei as Pei.
ETA: Although also yes, his homepage suggests that he may be okay with being addressed as Doctor. I still advocate the general strategy of avoiding titles altogether, and if you do use titles, refer to Professors as Professors (failure to do so will not offend anyone, but may make you look silly).
The situation in the US and Canada is quite relaxed, actually, nothing like in, say, Germany. Dr is a perfectly valid form of address to any faculty member.
Well, at least in my experience the Professors who don’t actually have doctorates tend not to appreciate having to correct you on that point. But yeah.
When I received the proofs for my IJMC papers, the e-mail addressed me as “dear professor Sotala” (for those who aren’t aware, I don’t even have a Master’s degree, let alone a professorship). When I mentioned this on Facebook, some people mentioned that there are countries where it’s a huge faux pas to address a professor as anything else than a professor. So since “professor” is the highest form of address, everyone tends to get called that in academic communication, just to make sure that nobody’ll be offended—even if the sender is 95% sure that the other isn’t actually a professor.
I really would not have guessed that it would be considered polite or appropriate to call someone a “higher” form of address than they’re entitled to, especially when it actually refers to something concrete. Learn something new every day, I guess.
I think AI is dangerous, that making safe AI is difficult, and that SI will likely fail in their mission. I donate to them in the hopes that this improves their chances.
I was dismayed that Pei has such a poor opinion of the Singularity Institute’s arguments, and that he thinks we are not making a constructive contribution. If we want the support of the AGI community, it seems we’ll have to improve our communication.
It might be more worthwhile to try to persuade graduate students and undergraduates who might be considering careers in AI research, since the personal cost associated with deciding that AI research is dangerous is lower for them. So less motivated cognition.
Correct me if I’m wrong, but isn’t it the case that you wish to decelerate AI research ? In this case, you are in fact making a destructive contribution—from the point of view of someone like Wang, who is interested in AI research. I see nothing odd about that.
It sounds as though you mean decelerating the bits that he is interested in and accelerating the bits that the SI is interested in. Rather as though the SI is after a bigger slice of the pie.
If you slow down capability research, then someone else is likely to become capable before you—in which case, your “goal management research” may not be so useful. How confident are you that this is a good idea?
If we want the support of the AGI community, it seems we’ll have to improve our communication.
Yes, this does seem to be an issue. When people in academia write something like “The “friendly AI” approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most AGI researchers. The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.”, the communication must be at an all-time low.
The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.”, the communication must be at an all-time low.
Well, of course. Imagine Eliezer would have founded SI to deal with physical singularities as a result of high-energy physics experiments. Would anything that he has written convince the physics community to listen to him? No, because he simply hasn’t written enough about physics to either convince them that he knows what he is talking about or to make his claims concrete enough to be critized in the first place.
Yet he has been more specific when it comes to physics than AI. So why would the AGI community listen to him?
I wouldn’t be as worried if they took it upon themselves to study AI risk independently, but rather than “not listen to Eliezer”, the actual event seems to be “not pay attention to AI risks” as a whole.
I wouldn’t be as worried if they took it upon themselves to study AI risk independently, but rather than “not listen to Eliezer”, the actual event seems to be “not pay attention to AI risks” as a whole.
Think about it this way. There are a handful of people like Jürgen Schmidhuber who share SI’s conception of AGI and its potential. But most AI researchers, including Pei Wang, do not buy the idea of AGI’s that can quickly and vastly self-improve themselves to the point of getting out of control.
Telling most people in the AI community about AI risks is similar to telling neuroscientists that their work might lead to the creation of a society of uploads which will copy themselves millions of times and pose a risk due to the possibility of a value drift. What reaction do you anticipate?
Telling most people in the AI community about AI risks is similar to telling neuroscientists that their work might lead to the creation of a society of uploads which will copy themselves millions of times and pose a risk due to the possibility of a value drift. What reaction do you anticipate?
One neuroscientist thought about it for a while, then said “yes, you’re probably right”. Then he co-authored with me a paper touching upon that topic. :-)
One neuroscientist thought about it for a while, then said “yes, you’re probably right”. Then he co-authored with me a paper touching upon that topic. :-)
Awesome reply. Which of your papers around this subject is the one with the co-author? (ie. Not so much ‘citation needed’ as ‘citation would have really powered home the point there!’)
But most AI researchers, including Pei Wang, do not buy the idea of AGI’s that can quickly and vastly self-improve themselves to the point of getting out of control.
To rephrase into a positive belief statement: most AI researches, including Pei Wang, believe that AGI’s are safely controllable.
Telling most people in the AI community about AI risks is similar to telling neuroscientists that their work might lead to the creation of a society of uploads which will copy themselves millions of times and pose a risk due to the possibility of a value drift. What reaction do you anticipate?
“Really? Awesome! Let’s get right on that.” (ref. early Eliezer)
Alternatively: ” Hmm? Yes, that’s interesting… it doesn’t apply to my current grant / paper, so… .”
“Really? Awesome! Let’s get right on that.” (ref. early Eliezer)
Alternatively: ” Hmm? Yes, that’s interesting… it doesn’t apply to my current grant / paper, so… .”
I didn’t expect that you would anticipate that. What I anticipate is outright ridicule of such ideas outside of science fiction novels. At least for most neuroscientists.
But most AI researchers, including Pei Wang, do not buy the idea of AGI’s that can quickly and vastly self-improve themselves to the point of getting out of control.
Well, that happening doesn’t seem terribly likely. That might be what happens if civilization is daydreaming during the process—but there’s probably going to be a “throttle”—and it will probably be carefully monitored—precisely in order to prevent anything untoward from happening.
I think you must first consider simpler possibility that SIAI actually has a very bad argument, and isn’t making any positive contribution to saving mankind from anything. When you have very good reasons to think it isn’t so (high iq test scores don’t suffice), very well verified given all the biases, you can consider possibility that it is miscommunication.
As I said about a previous discussion with Ben Goertzel, they seem to agree quite a bit about the dangers, but not about how much the Singularity Institute might affect the outcome.
If one were to phrase it differently, it might be, “Yes, AIs are incredibly, world-threateningly dangerous, but really, there’s nothing you can do about it.”
The cryonics approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most people. People have ignored it, not because it is indisputable, but because people have not bothered to criticize it.
Edit: Yeah, this was meant as a quote.
The question is whether “AGI researchers” are experts on “AI safety”. If the answer is “yes”, we should update in their direction simply because they are experts. But if the situation is like mine, then Pei Wang is committing argumentum ad populum. Not only should we not pay attention, we should point this out to him.
Point (4) of the first reply from Pei Wang. I didn’t noticed, but there are other deviations from the original phrasing, to eliminate direct references to the AGI community. It merely refers to “people” instead, making it a bit of a straw man. Charles’ point may still stand however, if most of the medical profession thinks cryonics doesn’t work (meaning, it is a false hope).
To make a quote, put a “>” at the beginning of the first line of the paragraph, like you would in an e-mail:
LessWrong is based on Reddit code, which uses Markdown syntax. It’s based on email conventions. Clik on the “Show Help button” at the bottom-right of your editing window when you write a comment, it’s a good quick reference.
LessWrong is based on Reddit code, which uses Markdown syntax. It’s based on email conventions. Clik on the “Show Help button” at the bottom-right of your editing window when you write a comment, it’s a good quick reference.
Your introduction style is flawless. I was expecting either a daringfireball link or a mention of the ‘Help’ link but you have included both as well as a given the history and explained the intuitive basis.
I hope you’ll pardon me for playing along a little there. It was a novel experience to be the one receiving the quoting instructions rather than the one typing them out. I liked the feeling of anonymity it gave me and wanted to see if that anonymity could be extended as far as acting out the role of a newcomer seeking further instructions.
Pleased to meet you loup-vaillant and thank you for making my counterfactual newcomer self feel welcome!
You got me. Overall, I preffer to judge posts by their content, so I’m glad to learn of your trick.
For the record, even I expected to stop at the Daring Fireball link. I also wrote a bunch of examples, but only then noticed/remembered the “Show help” button. I also erased a sentence about how to show markdown code in markdown (it’s rarely useful here, there was the Daring Fireball link, and my real reason for writing it was to show off).
I tend to heavily edit my writings. My most useful heuristic so far is “shorter is better”. This very comment benefited from it (let’s stop the recursion right there).
First, thank you for publishing this illuminating exchange.
I must say that Pei Wang sounds way more convincing to an uninitiated, but curious and mildly intelligent lay person (that would be me). Does not mean he is right, but he sure does make sense.
When Luke goes on to make a point, I often get lost in a jargon (“manifest convergent instrumental goals”) or have to look up a paper that Pei (or other AGI researchers) does not hold in high regard. When Pei Wang makes an argument, it is intuitively clear and does not require going through a complex chain of reasoning outlined in the works of one Eliezer Yudkowsky and not vetted by the AI community at large. This is, of course, not a guarantee of its validity, but it sure is easier to follow.
Some of the statements are quite damning, actually: “The “friendly AI” approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most AGI researchers. The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.” If one were to replace AI with physics, I would tend to dismiss EY as a crank just based on this statement, assuming it is accurate.
What makes me trust Pei Wang more than Luke is the common-sense statements like “to make AGI safe, to control their experience will probably be the main approach (which is what “education” is all about), but even that cannot guarantee safety.” and “unless you get a right idea about what AGI is and how it can be built, it is very unlikely for you to know how to make it safe”. Similarly, the SIAI position of “accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI first, rather than arbitrary superhuman AGI” rubs me the wrong way. While it does not necessarily mean it is wrong, the inability to convince outside experts that it is right is not a good sign.
This might be my confirmation bias, but I would be hard pressed to disagree with “To develop a non-trivial education theory of AGI requires a good understanding about how the system works, so if we don’t know how to build an AGI, there is no chance for us to know how to make it safe. I don’t think a good education theory can be “proved” in advance, pure theoretically. Rather, we’ll learn most of it by interacting with baby AGIs, just like how many of us learn how to educate children.”
As a side point, I cannot help but wonder if the outcome of this discussion would have been different were it EY and not LM involved in it.
This sort of “common sense” can be highly misleading! For example, here Wang is drawing parallels between a nascent AI and a human child to argue about nature vs nurture. But if we compare a human and a different social animal, we’ll see that most of the differences in their behavior are innate and the gap can’t be covered by any amount of “education”: e.g. humans can’t really become as altruistic and self-sacrificing as worker ants because they’ll still retain some self-preservation instinct, no matter how you brainwash them.
What makes Wang think that this sort of fixed attitude—which can be made more hard-wired than the instincts of biological organisms—cannot manifest itself in an AGI?
(I’m certain that a serious AI thinker, or just someone with good logic and clear thinking, could find a lot more holes in such “common sense” talk.)
Presumably the argument is something like:
You can’t build an AI that is intelligent from the moment you switch it on: you have to train it.
We know how to train intelligence into humans, its called education
An AI that lacked human-style instincts and learning abilities at switch-on wouldn’t be trainable by us, we just wouldn’t know how, so it would never reach intelligence.
I expect Eliezer to have displayed less patience than Luke did (a more or less generalizable prediction.)
I felt the main reason was anthropomorphism:
Note that I don’t want to accuse Pei Wang of anthropomorphism. My point is, his choice of words appeal to our anthropomorphism, which is highly intuitive. Another example of an highly intuitive, but not very helpful sentence:
Intuitive, because applied to humans, we can easily see that we can change plans according to experience. Like apply for a PhD, then dropping out when finding out you don’t enjoy it after all. You can abandon the goal of making research, and have a new goal of, say, practicing and teaching surfing.
Not very helpful, because the split between initial goals and later goals does not help you build an AI that will actually do something “good”. Here, the split between instrumental goals (means to an end), and terminal goals (the AI’s “ulterior motives”) is more important. To give a human example, in the case above, doing research or surfing are both means to the same end (like being happy, or something more careful but so complicated nobody knows how to clearly specify it yet). For an AI, as Pei Wang implies, the initial goals aren’t necessarily supposed to constraint all future goals. But its terminal goals are indeed supposed to constrain the instrumental goals it will form later. (More precisely, the instrumental goals are supposed to follow from the terminal goals and the AI’s current model of the world.)
Edit: it just occurred to me, that terminal goals have somehow to be encoded into the AI before we set it loose. They are necessarily initial goals (if they aren’t, the AI is by definition unfriendly —not a problem if its goals miraculously converge towards something “good”, though). Thinking about it, it looks like Pei Wang doesn’t believe it is possible to make an AI with stable terminal goals.
Excellent Freudian slip there.
Corrected, thanks.
I think I am in the same position as you are (uninitiated but curious) and I had the same immediate reaction that Pei was more convincing. However, for me, I think this was the result of two factors
Pei is a Professor
Pei treated the interview like a conversation with someone who has read a couple books and that’s about it.
Maybe the 2nd point isn’t entirely true, but that was what immediately stuck out after thinking about why I was drawn to Pei’s arguments. Once I eliminated his status as a barometer for his arguments… it just became (1) an issue of my own lack of knowledge and (2) the tone of the responses.
For one thing, why the hell should I understand this in the first place? This is a dialogue between two prominent AI researchers. What I would expect from such a dialogue would be exactly what I would expect from sitting in on a graduate philosophy seminar or a computer science colloquium—I would be able to follow the gist of it, but not the gritty details. I would expect to hear some complex arguments that would require a couple textbooks and a dozen tabs open in my browser to be able to follow.
But I was able to understand Pei’s arguments and play with them! If solving these kinds of conceptual problems is this easy, I might try to take over the world myself.
Not to say that the appearance of “complexity” is necessary for a good argument (EY’s essays are proof), but here it seems like this lack of complexity (or as someone else said, the appeal to common sense) is a warning for the easily persuaded. Rereading with these things in mind illuminates the discussion a bit better.
I was actually a bit depressed by this dialogue. It seemed like an earnest (but maybe a little over the top with the LW references) attempt by lukeprog to communicate interesting ideas. I may be setting my expectations a little high, but Pei seemed to think he was engaging an undergraduate asking about sorting algorithms.
Of course, I could be completely misinterpreting things. I thought I would share my thought process after I came to the same conclusion as you did.
And he thought the undergrad terribly naive for not understanding that all sorting algorithms are actually just bubble sort.
This is why I find that unless the individual is remarkably open—to the point of being peculiar—it is usually pointless to try to communicate across status barriers. Status makes people (have the tendency and social incentives that make them act) stupid when it comes to comprehending others.
That’s an incredibly sweeping statement. Are all pop-sci publications useless?
Reference.
Do you think that generalises to academics? Wouldn’t a researcher who never changed their mind about anything be dismissed as a hidebound fogey?
What? This was a dialog between Pei and lukeprog, right?
I’m curious about what you mean by the appellation “prominent AI researcher” that you would apply it to lukeprog, and whether he considers himself as a member of that category.
Um… but these are statements I agreed with.
I wish Pei had taken the time to read the articles I repeatedly linked to, for they were written precisely to explain why his position is misguided.
I think you should have listed a couple of the most important articles at the beginning as necessary background reading to understand your positions and terminology (like Pei did with his papers), and then only used links very sparingly afterwards. Unless you already know your conversation partner takes you very seriously, you can’t put 5 hyperlinks in an email and expect the other person to read them all. When they see that many links, they’ll probably just ignore all of them. (Not to mention the signaling issues that others already pointed out.)
The reactions I got (from a cognitive scientist and another researcher) is that Bostrom is a “sloppy thinker” (original words) and that SI’s understanding of AGI is naive.
Michael Littman told me he is going to read some of the stuff too. I haven’t got an answer yet though.
Hmm, maybe it is possible to summarize them in a language that an AI expert would find both meaningful and convincing. How is your mental model of Dr Wang?
Nitpick, but it’s Professor Wang, not Doctor Wang.
The page linked at the top of the article says Dr. Wang. And his CV says he’s a Ph.D.
The title of Professor supersedes the title of Doctor, at least in the case of a PhD (I’m not sure about MD, but would assume similarly). His CV indicates pretty clearly that he is an Associate Professor at temple university, so the correct title is Professor.
Again, I am being somewhat super-pedantic here, and I apologize for any annoyance this causes. But hopefully it will help you in your future signalling endeavors.
Also, in most situations it is okay to just go by first name, or full name (without any titles); I have I think exclusively referred to Pei as Pei.
ETA: Although also yes, his homepage suggests that he may be okay with being addressed as Doctor. I still advocate the general strategy of avoiding titles altogether, and if you do use titles, refer to Professors as Professors (failure to do so will not offend anyone, but may make you look silly).
...Not in my experience. Do you have some particular reason to believe this is the case in Philadelphia?
The situation in the US and Canada is quite relaxed, actually, nothing like in, say, Germany. Dr is a perfectly valid form of address to any faculty member.
Well, at least in my experience the Professors who don’t actually have doctorates tend not to appreciate having to correct you on that point. But yeah.
When I received the proofs for my IJMC papers, the e-mail addressed me as “dear professor Sotala” (for those who aren’t aware, I don’t even have a Master’s degree, let alone a professorship). When I mentioned this on Facebook, some people mentioned that there are countries where it’s a huge faux pas to address a professor as anything else than a professor. So since “professor” is the highest form of address, everyone tends to get called that in academic communication, just to make sure that nobody’ll be offended—even if the sender is 95% sure that the other isn’t actually a professor.
I really would not have guessed that it would be considered polite or appropriate to call someone a “higher” form of address than they’re entitled to, especially when it actually refers to something concrete. Learn something new every day, I guess.
Yeah, it was pretty easy for me to nod my head along with most of it, pointing to my “SI failure mode” bucket.
Please clarify.
I think AI is dangerous, that making safe AI is difficult, and that SI will likely fail in their mission. I donate to them in the hopes that this improves their chances.
I found this reaction enlightening. Thanks for writing it up.
What is your reaction?
I was dismayed that Pei has such a poor opinion of the Singularity Institute’s arguments, and that he thinks we are not making a constructive contribution. If we want the support of the AGI community, it seems we’ll have to improve our communication.
It might be more worthwhile to try to persuade graduate students and undergraduates who might be considering careers in AI research, since the personal cost associated with deciding that AI research is dangerous is lower for them. So less motivated cognition.
“It is difficult to get a man to understand something, when his salary depends upon his not understanding it”—Upton Sinclair
Good point!
Correct me if I’m wrong, but isn’t it the case that you wish to decelerate AI research ? In this case, you are in fact making a destructive contribution—from the point of view of someone like Wang, who is interested in AI research. I see nothing odd about that.
To decelerate AI capability research and accelerate AI goal management research. An emphasis shift, not a decrease. An increase would be in order.
It sounds as though you mean decelerating the bits that he is interested in and accelerating the bits that the SI is interested in. Rather as though the SI is after a bigger slice of the pie.
If you slow down capability research, then someone else is likely to become capable before you—in which case, your “goal management research” may not be so useful. How confident are you that this is a good idea?
Yes, this does seem to be an issue. When people in academia write something like “The “friendly AI” approach advocated by Eliezer Yudkowsky has several serious conceptual and theoretical problems, and is not accepted by most AGI researchers. The AGI community has ignored it, not because it is indisputable, but because people have not bothered to criticize it.”, the communication must be at an all-time low.
Well, of course. Imagine Eliezer would have founded SI to deal with physical singularities as a result of high-energy physics experiments. Would anything that he has written convince the physics community to listen to him? No, because he simply hasn’t written enough about physics to either convince them that he knows what he is talking about or to make his claims concrete enough to be critized in the first place.
Yet he has been more specific when it comes to physics than AI. So why would the AGI community listen to him?
I wouldn’t be as worried if they took it upon themselves to study AI risk independently, but rather than “not listen to Eliezer”, the actual event seems to be “not pay attention to AI risks” as a whole.
Think about it this way. There are a handful of people like Jürgen Schmidhuber who share SI’s conception of AGI and its potential. But most AI researchers, including Pei Wang, do not buy the idea of AGI’s that can quickly and vastly self-improve themselves to the point of getting out of control.
Telling most people in the AI community about AI risks is similar to telling neuroscientists that their work might lead to the creation of a society of uploads which will copy themselves millions of times and pose a risk due to the possibility of a value drift. What reaction do you anticipate?
One neuroscientist thought about it for a while, then said “yes, you’re probably right”. Then he co-authored with me a paper touching upon that topic. :-)
(Okay, probably not a very typical case.)
Awesome reply. Which of your papers around this subject is the one with the co-author? (ie. Not so much ‘citation needed’ as ‘citation would have really powered home the point there!’)
Edited citations to the original comment.
To rephrase into a positive belief statement: most AI researches, including Pei Wang, believe that AGI’s are safely controllable.
“Really? Awesome! Let’s get right on that.” (ref. early Eliezer)
Alternatively: ” Hmm? Yes, that’s interesting… it doesn’t apply to my current grant / paper, so… .”
I didn’t expect that you would anticipate that. What I anticipate is outright ridicule of such ideas outside of science fiction novels. At least for most neuroscientists.
Sure, that too.
Well, that happening doesn’t seem terribly likely. That might be what happens if civilization is daydreaming during the process—but there’s probably going to be a “throttle”—and it will probably be carefully monitored—precisely in order to prevent anything untoward from happening.
Hey Tim, you can create another AI safety nonprofit to make sure things happen that way!
;-)
Seriously, I will donate!
Poor analogy. Physicists considered this possibility carefully and came up a superfluity of totally airtight reasons to dismiss the concern.
I think you must first consider simpler possibility that SIAI actually has a very bad argument, and isn’t making any positive contribution to saving mankind from anything. When you have very good reasons to think it isn’t so (high iq test scores don’t suffice), very well verified given all the biases, you can consider possibility that it is miscommunication.
This may provide more data on what “the AGI community” thinks:
http://wiki.lesswrong.com/wiki/Interview_series_on_risks_from_AI
As I said about a previous discussion with Ben Goertzel, they seem to agree quite a bit about the dangers, but not about how much the Singularity Institute might affect the outcome.
If one were to phrase it differently, it might be, “Yes, AIs are incredibly, world-threateningly dangerous, but really, there’s nothing you can do about it.”
Edit: Yeah, this was meant as a quote.
The question is whether “AGI researchers” are experts on “AI safety”. If the answer is “yes”, we should update in their direction simply because they are experts. But if the situation is like mine, then Pei Wang is committing argumentum ad populum. Not only should we not pay attention, we should point this out to him.
(You may want to put “cryonics” between square brackets, I nearly missed this deviation from the original quote.)
The grandparent is a quote? That probably should be indicated somehow. I was about to reply as if it was simply his words.
Point (4) of the first reply from Pei Wang. I didn’t noticed, but there are other deviations from the original phrasing, to eliminate direct references to the AGI community. It merely refers to “people” instead, making it a bit of a straw man. Charles’ point may still stand however, if most of the medical profession thinks cryonics doesn’t work (meaning, it is a false hope).
To make a quote, put a “
>” at the beginning of the first line of the paragraph, like you would in an e-mail:Oh, it’s that simple? How do you find this sort of thing out?
LessWrong is based on Reddit code, which uses Markdown syntax. It’s based on email conventions. Clik on the “Show Help button” at the bottom-right of your editing window when you write a comment, it’s a good quick reference.
Your introduction style is flawless. I was expecting either a daringfireball link or a mention of the ‘Help’ link but you have included both as well as a given the history and explained the intuitive basis.
I hope you’ll pardon me for playing along a little there. It was a novel experience to be the one receiving the quoting instructions rather than the one typing them out. I liked the feeling of anonymity it gave me and wanted to see if that anonymity could be extended as far as acting out the role of a newcomer seeking further instructions.
Pleased to meet you loup-vaillant and thank you for making my counterfactual newcomer self feel welcome!
You got me. Overall, I preffer to judge posts by their content, so I’m glad to learn of your trick.
For the record, even I expected to stop at the Daring Fireball link. I also wrote a bunch of examples, but only then noticed/remembered the “Show help” button. I also erased a sentence about how to show markdown code in markdown (it’s rarely useful here, there was the Daring Fireball link, and my real reason for writing it was to show off).
I tend to heavily edit my writings. My most useful heuristic so far is “shorter is better”. This very comment benefited from it (let’s stop the recursion right there).
It seemed gentler than responding with a direct challenge to the inference behind the presumption.