Agreed—AGI will probably not be developed with the aim of improving science.
I also want to quibble about this:
Therefore the next logical step is to use science to figure out how to replace humans by a better version of themselves
Since most readers don’t want to be replaced, at least in one interpretation of that term, this line sticks in the throat and breaks the flow. The natural response is something like “logical? According to whose goals?”
Agreed—AGI will probably not be developed with the aim of improving science.
I also want to quibble about this:
Since most readers don’t want to be replaced, at least in one interpretation of that term, this line sticks in the throat and breaks the flow. The natural response is something like “logical? According to whose goals?”