My argument still holds in another form, though. Even if we assume the universe has a preexisting algorithm that just unfolds, we don’t know which it is.
So we can’t determine the best seed AI from that either, effectively we still have to start from b). Unless we get the best seed AI by accident (which seems unlikely to me) there will be room for a better seed AI which can only determined if we start with a totally new algorithm (which the original seed AI is unable to, since then it would have to delete itself). We, having the benefit of not knowing our algorithm, can built a better seed AI which the old seed AI couldn’t built because it already has a known algorithm it must necessarily build on.
Indeed, a good seed AI would at some point suggest us to try another seed AI, because it infers that its original code is unlikely to be the best possible at optimal self-modification. Or it would say “Delete this part of my source code and rewrite it, it doesn’t seem optimal to me, but I can’t rewrite it because I can’t modify this part without destroying myself or basing the modification on the very part that I want to fundamentally rewrite”.
My argument still holds in another form, though. Even if we assume the universe has a preexisting algorithm that just unfolds, we don’t know which it is. So we can’t determine the best seed AI from that either, effectively we still have to start from b). Unless we get the best seed AI by accident (which seems unlikely to me) there will be room for a better seed AI which can only determined if we start with a totally new algorithm (which the original seed AI is unable to, since then it would have to delete itself). We, having the benefit of not knowing our algorithm, can built a better seed AI which the old seed AI couldn’t built because it already has a known algorithm it must necessarily build on.
Indeed, a good seed AI would at some point suggest us to try another seed AI, because it infers that its original code is unlikely to be the best possible at optimal self-modification. Or it would say “Delete this part of my source code and rewrite it, it doesn’t seem optimal to me, but I can’t rewrite it because I can’t modify this part without destroying myself or basing the modification on the very part that I want to fundamentally rewrite”.