Yes, good point. The objection is about SI not addressing tool-AI, much of their discussion is about addressing tool-AI, not the meta “why isn’t this explicitly called out by SI?” In particular the intuitions Holden has as responses to those questions, that we may well be able to create extremely useful general AI without creating general AI that can improve itself, do seem like they have received too little in-depth discussion here. We’ve often mentioned the possibility and often decided to skip the question because it’s very hard to think about but I don’t recall many really lucid conversations trying to ferret out what it would look like if more-than-narrow, less-than-having-the-ability-to-self-improve AI were a large enough target to reliably hit.
(as an aside, I think as Holden thinks of it, tool-AI could self improve, but because it’s tool like and not agent-like, it would not automatically self improve. Its outputs could be of the form “I would decide to rewrite my program with code X”, but humans would need to actually implement these changes.)
Yes, good point. The objection is about SI not addressing tool-AI, much of their discussion is about addressing tool-AI, not the meta “why isn’t this explicitly called out by SI?” In particular the intuitions Holden has as responses to those questions, that we may well be able to create extremely useful general AI without creating general AI that can improve itself, do seem like they have received too little in-depth discussion here. We’ve often mentioned the possibility and often decided to skip the question because it’s very hard to think about but I don’t recall many really lucid conversations trying to ferret out what it would look like if more-than-narrow, less-than-having-the-ability-to-self-improve AI were a large enough target to reliably hit.
(as an aside, I think as Holden thinks of it, tool-AI could self improve, but because it’s tool like and not agent-like, it would not automatically self improve. Its outputs could be of the form “I would decide to rewrite my program with code X”, but humans would need to actually implement these changes.)