I have similar misgivings, they prompted me to write the post. Fighting fire with fire looks like a dangerous idea. The problem statement should look like “how do we stop unfriendly AIs”, not “how do we make friendly AIs”. Many people here (e.g. Nesov and SarahC) seem convinced that the latter is the most efficient way of achieving the former. I hope we can find a better way if we think some more.
The problem statement should look like “how do we stop unfriendly AIs”, not “how do we make friendly AIs”.
If the universe is capable of running super-intelligent beings, then eventually either there will be one, or civilization will collapse. Maintaining the current state where there are no minds more intelligent than base humans seems very unlikely to be stable in the long run.
Given that, it seems the problem should be framed as “how do we end up with a super-intelligent being (or beings) that will go on to rearrange the universe the way we prefer?” which is not too different from “how do we make friendly AIs” if we interpret things like recursively-improved uploads as AIs.
I have similar misgivings, they prompted me to write the post. Fighting fire with fire looks like a dangerous idea. The problem statement should look like “how do we stop unfriendly AIs”, not “how do we make friendly AIs”. Many people here (e.g. Nesov and SarahC) seem convinced that the latter is the most efficient way of achieving the former. I hope we can find a better way if we think some more.
If the universe is capable of running super-intelligent beings, then eventually either there will be one, or civilization will collapse. Maintaining the current state where there are no minds more intelligent than base humans seems very unlikely to be stable in the long run.
Given that, it seems the problem should be framed as “how do we end up with a super-intelligent being (or beings) that will go on to rearrange the universe the way we prefer?” which is not too different from “how do we make friendly AIs” if we interpret things like recursively-improved uploads as AIs.