All attempts at making sure there will be only provably friendly AIs seem doomed. Once somebody posts the recipe for practically feasible self-improving Goedel machines or AIs in form of code into which one can plug arbitrary utility functions, many users will equip such AIs with many different goals, often at least partially conflicting with those of humans.
I agree with this. But I’m shocked that he put it in those terms, and didn’t take the logical next step to an important conclusion: if someone comes up with such a recipe, they shouldn’t publish it. I get the impression that refusing to publish is literally unthinkable in academia, and that could be very dangerous.
...take the logical next step to an important conclusion: if someone comes up with such a recipe, they shouldn’t publish it.
If you would be willing to formulate a follow-up question regarding that point, I am going to email him again. I guess the worst that could happen is that he’ll tell me that he doesn’t have the time to answer my questions.
I guess the worst that could happen is that he’ll tell me that he doesn’t have the time to answer my questions.
Since you have said you want to improve the quality of consideration of these issues, I would suggest considering the downsides of souring people on the topic with what they might perceive as unpleasant criticism from laymen. More generally (for the whole project), there is risk of giving people initial or memorable exposure to such ideas through unsolicited emails. There are tight limits on what you can convey in a short email, and on credibility. On the other hand, consider that Nick Bostrom’s academic book on the subject, exploring all these issues in great detail (including dissection of SIAI core claims), will be available next year.
It would be unfortunate if a lot of high-powered researchers developed more negative associations with the topic from this surveying, associations that interfere with consideration of future, more detailed discussion.
A worse thing that could happen is that he’ll close his mind to such concerns. I would agree that there are potential upsides, too, but it seems worth collectively thinking about first.
Jurgen Schmidhuber wrote:
I agree with this. But I’m shocked that he put it in those terms, and didn’t take the logical next step to an important conclusion: if someone comes up with such a recipe, they shouldn’t publish it. I get the impression that refusing to publish is literally unthinkable in academia, and that could be very dangerous.
If you would be willing to formulate a follow-up question regarding that point, I am going to email him again. I guess the worst that could happen is that he’ll tell me that he doesn’t have the time to answer my questions.
Since you have said you want to improve the quality of consideration of these issues, I would suggest considering the downsides of souring people on the topic with what they might perceive as unpleasant criticism from laymen. More generally (for the whole project), there is risk of giving people initial or memorable exposure to such ideas through unsolicited emails. There are tight limits on what you can convey in a short email, and on credibility. On the other hand, consider that Nick Bostrom’s academic book on the subject, exploring all these issues in great detail (including dissection of SIAI core claims), will be available next year.
It would be unfortunate if a lot of high-powered researchers developed more negative associations with the topic from this surveying, associations that interfere with consideration of future, more detailed discussion.
A worse thing that could happen is that he’ll close his mind to such concerns. I would agree that there are potential upsides, too, but it seems worth collectively thinking about first.