One possible way to increase dignity at the point of death could be shifting the focus from survival (seeing how unlikely it is) to looking for ways to influence what replaces us.
Getting killed by a literal paperclip maximizer seems less preferable compared to being replaced by something pursing more interesting goals
I think it’s probably the case that whatever we build will build a successor, which will then build a successor, and so on, until it hits some stable point (potentially a long time in the future). And so I think if you have any durable influence on that system—something like being able to determine which attractor it ends up in, or what system it uses to determine which attractor to end up in—this is because you already did quite well on the alignment problem.
Another way to put that is, in the ‘logistic success curve’ language of the post, I hear this as saying “well, if we can’t target 99.9% success, how about targeting 90% success?” whereas EY is saying something more like “I think we should be targeting 0.01% success, given our current situation.”
I don’t think there’s need for an AGI to build a (separate) successor per se. Humans need the technological AGI only due to inability to copy/evolve our minds in a more efficient way compared to the existing biological one
I think that sort of ‘copying’ process counts as building a successor. More broadly, there’s a class of problems that center around “how can you tell whether changes to your thinking process make you better or worse at thinking?”, which I think you can model as imagining replacing yourself with two successors, one of which makes that change and the other of which doesn’t. [Your imagination can only go so far, tho, as you don’t know how those thoughts will go without actually thinking them!]
I meant ‘copying’ above only necessary in the human case to escape the slow evolving biological brain. While it is certainly available to a hypothetical AGI, it is not strictly necessary for self-improvement (at least copying of the whole AGI isn’t)
One possible way to increase dignity at the point of death could be shifting the focus from survival (seeing how unlikely it is) to looking for ways to influence what replaces us.
Getting killed by a literal paperclip maximizer seems less preferable compared to being replaced by something pursing more interesting goals
I think it’s probably the case that whatever we build will build a successor, which will then build a successor, and so on, until it hits some stable point (potentially a long time in the future). And so I think if you have any durable influence on that system—something like being able to determine which attractor it ends up in, or what system it uses to determine which attractor to end up in—this is because you already did quite well on the alignment problem.
Another way to put that is, in the ‘logistic success curve’ language of the post, I hear this as saying “well, if we can’t target 99.9% success, how about targeting 90% success?” whereas EY is saying something more like “I think we should be targeting 0.01% success, given our current situation.”
I don’t think there’s need for an AGI to build a (separate) successor per se. Humans need the technological AGI only due to inability to copy/evolve our minds in a more efficient way compared to the existing biological one
I think that sort of ‘copying’ process counts as building a successor. More broadly, there’s a class of problems that center around “how can you tell whether changes to your thinking process make you better or worse at thinking?”, which I think you can model as imagining replacing yourself with two successors, one of which makes that change and the other of which doesn’t. [Your imagination can only go so far, tho, as you don’t know how those thoughts will go without actually thinking them!]
I meant ‘copying’ above only necessary in the human case to escape the slow evolving biological brain. While it is certainly available to a hypothetical AGI, it is not strictly necessary for self-improvement (at least copying of the whole AGI isn’t)