(I am not one of the people who have expressed skepticism, but I find myself with what I take to be feelings somewhat similar to theirs.)
I agree with 1 if it success is defined rather strictly (e.g., requiring that one human brain contain all the information in a form that actually enables the person whose brain it is to play like the bot does) but not necessarily if it is defined more laxly (e.g., it’s enough if for any given decision the bot makes we have a procedure that pretty much always gives us a human-comprehensible explanation of why it made that decision, with explanations for different decisions always fitting into a reasonably consistent framework).
I have no idea about 2; I don’t think I’ve seen any nontrivial but plausibly true propositions of the form “It is possible to be confident that advanced AGI systems will not pose an existential threat without X”, but on the other hand I don’t think this justifies much confidence that any specific X is a thing we should be working on if we care about being able to be confident that advanced AGI systems will not pose an existential threat.
I agree with 3, but I think this is because it hasn’t been defined as explicitly as possible rather than because of some fundamental unclarity in the question. Accordingly, I think 4 is probably wrong.
I’m not sure whether 5 is true or not and suspect that the answer depends mostly on how you choose to define “know”. (Maybe go bots don’t know anything!) I’m pretty confident saying that today’s best go bots know who’s likely to win in many typical game positions, or whether a given move kills a particular group or not. Accordingly, I am inclined to disagree with 6, even though probably there are edge cases where it’s not clear whether a given bot “knows” a given thing or not.
I don’t know what “essentially being” means in 7; as written it looks wrong to me, but for some strong definitions of “know everything it knows” something close enough might be true. E.g., plausibly certain bits of the KataGo network are encoding things roughly along the lines of “at the location we’re looking at white has a ponnuki shape whose influence is not negated by other nearby black groups”; one could know much of what the bot knows by regarding the ponnuki shape as valuable and understanding that some nearby configurations make it less valuable; but if knowing everything the bot knows includes computing exactly how good a given configuration of stones is in this respect, then plausibly you could only do that by having the ability to do pretty much the exact calculation the KataGo network does. (Perhaps the fine details are not part of what it knows but merely implementation details; maybe one could operationalize that in terms of the existence of similar, and similarly strong, bots where the fine details are somewhat different—though I think that, as it stands, is a bit too simplistic. Or perhaps I could claim that “I” know what the bot knows if I understand the overall structure and have a computer file, or a book, containing the actual numbers. Something something Chinese Room something something systems reply something.)
(I find I can’t help remarking that the final proposition makes me want to imagine a paper by Thomas Nagel entitled “What is it like to be a bot?. And of course it turns out that various people have in fact written pieces with that title.)
(I am not one of the people who have expressed skepticism, but I find myself with what I take to be feelings somewhat similar to theirs.)
I agree with 1 if it success is defined rather strictly (e.g., requiring that one human brain contain all the information in a form that actually enables the person whose brain it is to play like the bot does) but not necessarily if it is defined more laxly (e.g., it’s enough if for any given decision the bot makes we have a procedure that pretty much always gives us a human-comprehensible explanation of why it made that decision, with explanations for different decisions always fitting into a reasonably consistent framework).
I have no idea about 2; I don’t think I’ve seen any nontrivial but plausibly true propositions of the form “It is possible to be confident that advanced AGI systems will not pose an existential threat without X”, but on the other hand I don’t think this justifies much confidence that any specific X is a thing we should be working on if we care about being able to be confident that advanced AGI systems will not pose an existential threat.
I agree with 3, but I think this is because it hasn’t been defined as explicitly as possible rather than because of some fundamental unclarity in the question. Accordingly, I think 4 is probably wrong.
I’m not sure whether 5 is true or not and suspect that the answer depends mostly on how you choose to define “know”. (Maybe go bots don’t know anything!) I’m pretty confident saying that today’s best go bots know who’s likely to win in many typical game positions, or whether a given move kills a particular group or not. Accordingly, I am inclined to disagree with 6, even though probably there are edge cases where it’s not clear whether a given bot “knows” a given thing or not.
I don’t know what “essentially being” means in 7; as written it looks wrong to me, but for some strong definitions of “know everything it knows” something close enough might be true. E.g., plausibly certain bits of the KataGo network are encoding things roughly along the lines of “at the location we’re looking at white has a ponnuki shape whose influence is not negated by other nearby black groups”; one could know much of what the bot knows by regarding the ponnuki shape as valuable and understanding that some nearby configurations make it less valuable; but if knowing everything the bot knows includes computing exactly how good a given configuration of stones is in this respect, then plausibly you could only do that by having the ability to do pretty much the exact calculation the KataGo network does. (Perhaps the fine details are not part of what it knows but merely implementation details; maybe one could operationalize that in terms of the existence of similar, and similarly strong, bots where the fine details are somewhat different—though I think that, as it stands, is a bit too simplistic. Or perhaps I could claim that “I” know what the bot knows if I understand the overall structure and have a computer file, or a book, containing the actual numbers. Something something Chinese Room something something systems reply something.)
(I find I can’t help remarking that the final proposition makes me want to imagine a paper by Thomas Nagel entitled “What is it like to be a bot?. And of course it turns out that various people have in fact written pieces with that title.)
I think I basically agree with all of this.