(1) Intelligence is an extendible method that enables software to satisfy human preferences.
(2) If human preferences can be satisfied by an extendible method, humans have the capacity to extend the method.
(3) Extending the method that satisfies human preferences will yield software that is better at satisfying human preferences.
(4) Magic happens.
(5) There will be software that can satisfy all human preferences perfectly but which will instead satisfy orthogonal preferences, causing human extinction.
This is deeply silly. The thing about arguing from definitions is that you can prove anything you want if you just pick a sufficiently bad definition. That definition of intelligence is a sufficiently bad definition.
EDIT:
To extend this rebuttal in more detail:
I’m going to accept the definition of ‘intelligence’ given above. Now, here’s a parallel argument of my own:
Entelligence is an extendible method for satisfying an arbitrary set of preferences that are not human preferences.
If these preferences can be satisfied by an extendible method, then the entelligent agent has the capacity to extend the method.
Extending the method that satisfies these non-human preferences will yield software that’s better at satisfying non-human preferences.
The inevitable happens.
There will be software that will satisfy non-human preferences, causing human extinction.
Now, I pose to you: how do we make sure that we’re making intelligent software, and not “entelligent” software, under the above definitions? Obviously, this puts us back to the original problem of how to make a safe AI.
The original argument is rhetorical slight of hand. The given definition of intelligence implicitly assumes that the problem doesn’t exist, and all AI’s will be safe, and then goes on to prove that all AIs will be safe.
I didn’t downvote, but besides the obvious explanation of people being anti-anti-AI risk, I’ve seen you write these sort of one-liner comments that express your antipathy towards AI risk and do nothing else. Some people probably feel that their time is being wasted, and some people probably find it improbable that you can simultaneously be thinking your own thoughts and agree with an entire index of critiques. On the other hand, I can see from your perspective that there is a selection effect favoring people who take AI risk seriously and that you might think it prudent to represent your position whenever you can.
Let’s just take one of his most recent critiques. It’s an uncharitable interpretation of the standard position on why AGIs will not automatically do what you mean. The reason that there is not already UFAI is that even though AIs don’t share our goals, they lack optimization power. If I can trivially discover a misinterpretation of the standard position, then that lowers my estimate that you are examining his arguments critically or engaging in this debate charitably, which is behavior that is subject to social punishment in this community.
http://kruel.co/2012/07/17/ai-risk-critiques-index/
Kruel’s critique sounded very convincing when I first read it.
This is deeply silly. The thing about arguing from definitions is that you can prove anything you want if you just pick a sufficiently bad definition. That definition of intelligence is a sufficiently bad definition.
EDIT:
To extend this rebuttal in more detail:
I’m going to accept the definition of ‘intelligence’ given above. Now, here’s a parallel argument of my own:
Entelligence is an extendible method for satisfying an arbitrary set of preferences that are not human preferences.
If these preferences can be satisfied by an extendible method, then the entelligent agent has the capacity to extend the method.
Extending the method that satisfies these non-human preferences will yield software that’s better at satisfying non-human preferences.
The inevitable happens.
There will be software that will satisfy non-human preferences, causing human extinction.
Now, I pose to you: how do we make sure that we’re making intelligent software, and not “entelligent” software, under the above definitions? Obviously, this puts us back to the original problem of how to make a safe AI.
The original argument is rhetorical slight of hand. The given definition of intelligence implicitly assumes that the problem doesn’t exist, and all AI’s will be safe, and then goes on to prove that all AIs will be safe.
It’s really, fundamentally silly.
They remain extraordinarily convincing/accurate to me.
EDIT: Downvotes? Really?
I didn’t downvote, but besides the obvious explanation of people being anti-anti-AI risk, I’ve seen you write these sort of one-liner comments that express your antipathy towards AI risk and do nothing else. Some people probably feel that their time is being wasted, and some people probably find it improbable that you can simultaneously be thinking your own thoughts and agree with an entire index of critiques. On the other hand, I can see from your perspective that there is a selection effect favoring people who take AI risk seriously and that you might think it prudent to represent your position whenever you can.
Let’s just take one of his most recent critiques. It’s an uncharitable interpretation of the standard position on why AGIs will not automatically do what you mean. The reason that there is not already UFAI is that even though AIs don’t share our goals, they lack optimization power. If I can trivially discover a misinterpretation of the standard position, then that lowers my estimate that you are examining his arguments critically or engaging in this debate charitably, which is behavior that is subject to social punishment in this community.