Let me see if I can deal with the “no true scotsman” line of attack.
The way that that fallacy might apply to what I wrote would be, I think, something like this:
MIRI says that a superintelligence might unpack a goal statement like “maximize human happiness” by perpetrating a Maverick Nanny attack on humankind, but Loosemore says that no TRUE superintelligence would do such a thing, because it would be superintelligent enough to realize that this was a ‘mistake’ (in some sense).
This would be a No True Scotsman fallacy, because the term “superintelligence” has been, in effect, redefined by me to mean “something smart enough not to do that”.
Now, my take on the NTS idea is that it cannot be used if there are substantive grounds for saying that there are two categories involved, rather than a real category and a fake category that is (for some unexplained reason) exceptional.
Example: Person A claims that a sea-slug caused the swimmer’s leg to be bitten off, but Person B argues that no “true” sea-slug would have done this. In this example, Person B is not using a No True Scotsman argument, because there are darned good reasons for supposing that sea-slugs cannot bite off the legs of swimmers.
So it all comes down to whether someone accused of NTS is inventing a ficticious category distinction (“true” versus “non-true” Scotsman) solely for the purposes of supporting their argument.
In my case, what I have argued is right up there with the sea-slug argument. What I have said, in effect, is that if we sit down and carefully think about the type of “superintelligence” that MIRI et al. put into their scenarios, and if we explore all the implications of what that hypothetical AI would have to be like, we quickly discover some glaring inconsistencies in their scenarios. The sea-slug, in effect, is supposed to have bitten through bone with a mouth made of mucous. And the sea-slug is so small it could not wrap itself around the swimmer’s leg. Thinking through the whole sea-slug scenario leads us into a mass of evidence indicating that the proposed scenario is nuts. Similarly, thinking through the implications of an AI that is so completely unable to handle context, that it can live with Grade A contradictions at the heart of its reasoning, leads us to a mass of unbelievable inconsistencies in the ‘intelligence’ of this supposed superintelligence.
So, where the discussion needs to be, in respect of the paper, is in the exact details of why the proposed SI might not be a meaningful hypothetical. It all comes down to a meticulous dissection of the mechanisms involved.
To conclude: sorry if I seemed to come down a little heavy on you in my first response. I wasn’t upset, it was just that the NTS critique had occurred before. In some of those previous cases the NTS attack was accompanied by language that strongly implied that I had not just committed an NTS fallacy, but that I was such an idiot that my idiocy was grounds for recommending to all not to even read the paper. ;-)
″… thinking through the implications of an AI that is so completely unable to handle context, that it can live with Grade A contradictions at the heart of its reasoning, leads us to a mass of unbelievable inconsistencies in the ‘intelligence’ of this supposed superintelligence.”
This is all at once concise, understandable, and reassuring. Thank you. I still wonder if we are accurately broadening the scope of defined “intelligence” out too far, but my wonder comes from gaps in my specific knowledge and not from gaps in your argument.
Let me see if I can deal with the “no true scotsman” line of attack.
The way that that fallacy might apply to what I wrote would be, I think, something like this:
MIRI says that a superintelligence might unpack a goal statement like “maximize human happiness” by perpetrating a Maverick Nanny attack on humankind, but Loosemore says that no TRUE superintelligence would do such a thing, because it would be superintelligent enough to realize that this was a ‘mistake’ (in some sense).
This would be a No True Scotsman fallacy, because the term “superintelligence” has been, in effect, redefined by me to mean “something smart enough not to do that”.
Now, my take on the NTS idea is that it cannot be used if there are substantive grounds for saying that there are two categories involved, rather than a real category and a fake category that is (for some unexplained reason) exceptional.
Example: Person A claims that a sea-slug caused the swimmer’s leg to be bitten off, but Person B argues that no “true” sea-slug would have done this. In this example, Person B is not using a No True Scotsman argument, because there are darned good reasons for supposing that sea-slugs cannot bite off the legs of swimmers.
So it all comes down to whether someone accused of NTS is inventing a ficticious category distinction (“true” versus “non-true” Scotsman) solely for the purposes of supporting their argument.
In my case, what I have argued is right up there with the sea-slug argument. What I have said, in effect, is that if we sit down and carefully think about the type of “superintelligence” that MIRI et al. put into their scenarios, and if we explore all the implications of what that hypothetical AI would have to be like, we quickly discover some glaring inconsistencies in their scenarios. The sea-slug, in effect, is supposed to have bitten through bone with a mouth made of mucous. And the sea-slug is so small it could not wrap itself around the swimmer’s leg. Thinking through the whole sea-slug scenario leads us into a mass of evidence indicating that the proposed scenario is nuts. Similarly, thinking through the implications of an AI that is so completely unable to handle context, that it can live with Grade A contradictions at the heart of its reasoning, leads us to a mass of unbelievable inconsistencies in the ‘intelligence’ of this supposed superintelligence.
So, where the discussion needs to be, in respect of the paper, is in the exact details of why the proposed SI might not be a meaningful hypothetical. It all comes down to a meticulous dissection of the mechanisms involved.
To conclude: sorry if I seemed to come down a little heavy on you in my first response. I wasn’t upset, it was just that the NTS critique had occurred before. In some of those previous cases the NTS attack was accompanied by language that strongly implied that I had not just committed an NTS fallacy, but that I was such an idiot that my idiocy was grounds for recommending to all not to even read the paper. ;-)
″… thinking through the implications of an AI that is so completely unable to handle context, that it can live with Grade A contradictions at the heart of its reasoning, leads us to a mass of unbelievable inconsistencies in the ‘intelligence’ of this supposed superintelligence.”
This is all at once concise, understandable, and reassuring. Thank you. I still wonder if we are accurately broadening the scope of defined “intelligence” out too far, but my wonder comes from gaps in my specific knowledge and not from gaps in your argument.