maybe a sufficiently advanced general intelligence should be able to recognize itself given some incomplete knowledge of itself
Yeah, that would be good, but we have no idea how to do it mathematically yet. Humans have intuitions that approximate that, but evolution probably didn’t give us a mechanism that’s correct in general, we’ll have to come up with the right math ourselves.
Let F be a function which takes m in M and phi in the set of formulas of L as inputs and outputs the interpretation of phi in M
The big problem is defining F. If you squint the right way, you can view Paul’s idea as a way of asking the AI to figure out F.
Yeah, that would be good, but we have no idea how to do it mathematically yet. Humans have intuitions that approximate that, but evolution probably didn’t give us a mechanism that’s correct in general, we’ll have to come up with the right math ourselves.
The big problem is defining F. If you squint the right way, you can view Paul’s idea as a way of asking the AI to figure out F.