It’s still not clear to me why having an AI that is capable of answering the question “How do we make a better version of you?” automatically kills humans. Presumably, when the AI says “Here’s the source code to a better version of me”, we’d still be able to read through it and make sure it didn’t suddenly rewrite itself to be an agent instead of a tool. We’re assuming that, as a tool, the AI has no goals per se and thus no motivation to deceive us into turning it into an agent.
That said, depending on what you mean by “effective”, perhaps the AI doesn’t even need to be able to answer questions like “How do we write a better version of you?”
For example, we find Google Maps to be very useful, even though if you asked Google Maps “How do we make a better version of Google Maps?” it would probably not be able to give the types of answers we want.
A tool-AI which was smarter than the smartest human, and yet which could not simply spit out a better version of itself would still probably be a very useful AI.
It’s still not clear to me why having an AI that is capable of answering the question “How do we make a better version of you?” automatically kills humans. Presumably, when the AI says “Here’s the source code to a better version of me”, we’d still be able to read through it and make sure it didn’t suddenly rewrite itself to be an agent instead of a tool. We’re assuming that, as a tool, the AI has no goals per se and thus no motivation to deceive us into turning it into an agent.
That said, depending on what you mean by “effective”, perhaps the AI doesn’t even need to be able to answer questions like “How do we write a better version of you?”
For example, we find Google Maps to be very useful, even though if you asked Google Maps “How do we make a better version of Google Maps?” it would probably not be able to give the types of answers we want.
A tool-AI which was smarter than the smartest human, and yet which could not simply spit out a better version of itself would still probably be a very useful AI.