I have written a bit about this previously. Your post talks about a single AI, but I think it’s also worth considering a multipolar scenario in which there are multiple competing AIs which face a tradeoff between more rapid self-improvement/propagation and value stability. One relevant factor might be whether value stability is achieved before (AI) space colonization become feasible—if not, it may be difficult to prevent the spread of AIs valuing maximally rapid expansion.
I have written a bit about this previously. Your post talks about a single AI, but I think it’s also worth considering a multipolar scenario in which there are multiple competing AIs which face a tradeoff between more rapid self-improvement/propagation and value stability. One relevant factor might be whether value stability is achieved before (AI) space colonization become feasible—if not, it may be difficult to prevent the spread of AIs valuing maximally rapid expansion.
I’ve added the tag https://www.lesswrong.com/tag/recursive-self-improvement to this post and the post of yours that didn’t already have it.