Of course, the fork here is whether the AI executing a “pivotal act” shuts itself down, or stays and oversees the subsequent developments.
If it “stays in charge”, at least in relation to the “pivotal act”, then it is going to do more than just a “pivotal act”, although the goals should be further developed in collaboration with humanity.
If it executes a “pivotal act” and shuts itself down, this is a very tall order (e.g. it cannot correct any problems which might subsequently emerge with that “pivotal act”, so we are asking for a very high level of perfection and foresight).
Of course, the fork here is whether the AI executing a “pivotal act” shuts itself down, or stays and oversees the subsequent developments.
If it “stays in charge”, at least in relation to the “pivotal act”, then it is going to do more than just a “pivotal act”, although the goals should be further developed in collaboration with humanity.
If it executes a “pivotal act” and shuts itself down, this is a very tall order (e.g. it cannot correct any problems which might subsequently emerge with that “pivotal act”, so we are asking for a very high level of perfection and foresight).