One more for the list of reasons why AGIs may be more poweful than humans: Self-transparency. If an AI makes a mistake and finds out later, it can do forensic analysis of its memories, and trust that those memories reflect its actual past mental states. Humans can’t do that very well, because our memory isn’t detailed or reliable enough, and usually contains only the output of our thought processes and not the procedural details. An AI, on the other hand, could be designed in such a way that it can perfectly reconstruct any past state, by storing occasional snapshots and replaying inputs. It wouldn’t have to wait for a real-world mistake to benefit, either; it could just as easily test itself on hypothetical scenarios, and even manipulate its own memories and inputs to make the tests realistic.
This is especially powerful when combined with duplication and editability. It can collect a test suite of puzzles and reference solutions, and have very good testing of self modifications. By running slightly different versions of itself on the same input, starting from the same state, it would bypass the uncontrolled variability that plagues human cognitive psychology.
One more for the list of reasons why AGIs may be more poweful than humans: Self-transparency. If an AI makes a mistake and finds out later, it can do forensic analysis of its memories, and trust that those memories reflect its actual past mental states. Humans can’t do that very well, because our memory isn’t detailed or reliable enough, and usually contains only the output of our thought processes and not the procedural details. An AI, on the other hand, could be designed in such a way that it can perfectly reconstruct any past state, by storing occasional snapshots and replaying inputs. It wouldn’t have to wait for a real-world mistake to benefit, either; it could just as easily test itself on hypothetical scenarios, and even manipulate its own memories and inputs to make the tests realistic.
This is especially powerful when combined with duplication and editability. It can collect a test suite of puzzles and reference solutions, and have very good testing of self modifications. By running slightly different versions of itself on the same input, starting from the same state, it would bypass the uncontrolled variability that plagues human cognitive psychology.