I enjoyed this and found it to be a surprising deconstruction of the goal of provably safe self-modification.
I think there is also a more general thrust toward reflectively consistent AI architectures, which has been quite fruitful in highlighting open problems. This could be justified in terms of self-modification (and probably has been in most cases), but also might stand on its own as a reasonable desideratum.
I’m not fully convinced on the “standards of reasoning can’t be outsourced” point.
As things stand, I don’t think there is a plausible story for how an AI which started out having uncertainty over theories in 1st-order logic (as has been discussed fairly extensively) could later come to conceive of the standard model for the natural numbers, and other such concepts which lack a finite or R.E. axiomatization in 1st-order logic (or in any effective logic). This is just Skolem’s paradox.
The best story which I can surmise is that the axioms of set theory may be accepted on pragmatic grounds (they allow convenient description of many useful entities). This would then allow the existence and uniqueness of the standard model to be proved relative to those axioms.
Actually, this isn’t so bad; I think I habitually give this explanation too little credit.
I’m concerned, though; my feeling is that there should be something more resolving Skolem’s paradox (a difference in how we perform probabilistic reasoning for 2nd-order entities as opposed to 1st-order). If there is something more, it seems possible that an AI would miss it (view it as human irrationality).
I enjoyed this and found it to be a surprising deconstruction of the goal of provably safe self-modification.
I think there is also a more general thrust toward reflectively consistent AI architectures, which has been quite fruitful in highlighting open problems. This could be justified in terms of self-modification (and probably has been in most cases), but also might stand on its own as a reasonable desideratum.
I’m not fully convinced on the “standards of reasoning can’t be outsourced” point.
As things stand, I don’t think there is a plausible story for how an AI which started out having uncertainty over theories in 1st-order logic (as has been discussed fairly extensively) could later come to conceive of the standard model for the natural numbers, and other such concepts which lack a finite or R.E. axiomatization in 1st-order logic (or in any effective logic). This is just Skolem’s paradox.
The best story which I can surmise is that the axioms of set theory may be accepted on pragmatic grounds (they allow convenient description of many useful entities). This would then allow the existence and uniqueness of the standard model to be proved relative to those axioms.
Actually, this isn’t so bad; I think I habitually give this explanation too little credit.
I’m concerned, though; my feeling is that there should be something more resolving Skolem’s paradox (a difference in how we perform probabilistic reasoning for 2nd-order entities as opposed to 1st-order). If there is something more, it seems possible that an AI would miss it (view it as human irrationality).