fair enough—this is my caution against the logic “I can think of a risk, therefore we need to worry about it!” It seems that SIAI is making the stronger claim that unfriendliness is very likely.
My personal view is that AI is very hard itself, and that working on, say, a computer that can do what a mouse can do is likely to take a long time, and is harmless but very interesting research. I don’t think we’re anywhere near a point when we need to shut down anybody’s current research.
Consider marginal utility. Many people are working on AI, machine learning, computational psychology, and related fields. Nobody is working on preference theory, formal understanding of our goals under reflection. If you want to do interesting research and if you have the background to advance either of those fields, do you think the world will be better off with you on the one side or on the other?
Maybe that’s true, but that’s a separate point. “Let’s work on preference theory so that it’ll be ready when the AI catches up” is one thing—tentatively, I’d say it’s a good idea. “Let’s campaign against anybody doing AI research” seems less useful (and less likely to be effective.)
fair enough—this is my caution against the logic “I can think of a risk, therefore we need to worry about it!” It seems that SIAI is making the stronger claim that unfriendliness is very likely.
My personal view is that AI is very hard itself, and that working on, say, a computer that can do what a mouse can do is likely to take a long time, and is harmless but very interesting research. I don’t think we’re anywhere near a point when we need to shut down anybody’s current research.
Consider marginal utility. Many people are working on AI, machine learning, computational psychology, and related fields. Nobody is working on preference theory, formal understanding of our goals under reflection. If you want to do interesting research and if you have the background to advance either of those fields, do you think the world will be better off with you on the one side or on the other?
Maybe that’s true, but that’s a separate point. “Let’s work on preference theory so that it’ll be ready when the AI catches up” is one thing—tentatively, I’d say it’s a good idea. “Let’s campaign against anybody doing AI research” seems less useful (and less likely to be effective.)