I mean (following Shmidhuber) that A’ is better than A. Of course if A transforms into A’ then A’ and A must be about equally good, but what we should really ask is “how much better is A then some less reflective approach?” Then we could imagine large improvements: for example, A could use the first fourth of its time doubling the speed of its proof searcher, the next eighth doubling it again, the next sixteenth doubling it again, and end up finding a better object level solver that would have taken much longer than the entire allotted time to discover using the original proof searcher. The argument I gave suggests that this can’t happen in any obvious way, so that A is actually only a factor of 2 faster than a particular non-reflective proof searcher, but that seems to be a different issue.
When I wrote “these self-modifications” I meant “these (self-modifications + justifications)”, as safety was a property of a justified self-modification. I’ll try and clarify.
If A can prove that AO always does better with more time, then A can modify into A’ and simultaneously commit to spending less time doing proof search. If it spends much more time doing proof search, it runs into the sort of issue I highlighted (consider the additional justified self-modifications it looks at; it can’t prove they are all safe, so it can’t prove that considering them is good). I could try and make this argument more clear/explicit.
Thanks, this whole part of your post is clearer to me now. I think the post would benefit from integrating these explanations to some degree into the text.
I mean (following Shmidhuber) that A’ is better than A. Of course if A transforms into A’ then A’ and A must be about equally good, but what we should really ask is “how much better is A then some less reflective approach?” Then we could imagine large improvements: for example, A could use the first fourth of its time doubling the speed of its proof searcher, the next eighth doubling it again, the next sixteenth doubling it again, and end up finding a better object level solver that would have taken much longer than the entire allotted time to discover using the original proof searcher. The argument I gave suggests that this can’t happen in any obvious way, so that A is actually only a factor of 2 faster than a particular non-reflective proof searcher, but that seems to be a different issue.
When I wrote “these self-modifications” I meant “these (self-modifications + justifications)”, as safety was a property of a justified self-modification. I’ll try and clarify.
If A can prove that AO always does better with more time, then A can modify into A’ and simultaneously commit to spending less time doing proof search. If it spends much more time doing proof search, it runs into the sort of issue I highlighted (consider the additional justified self-modifications it looks at; it can’t prove they are all safe, so it can’t prove that considering them is good). I could try and make this argument more clear/explicit.
Thanks, this whole part of your post is clearer to me now. I think the post would benefit from integrating these explanations to some degree into the text.