Note that cognition comes with no goals attached. Emotions is where the drive to do anything comes from. You allude to it in “contemplation of a possible choice or outcome”. My suspicion is that having a component not available to accurate introspection is essential to functioning. An agent (well, an algorithm) that can completely model itself (or, rather, model its identical copy, to mitigate the usual decision theory paradoxes), may end up in a state where there is no intrinsic meaning to anything, a nirvana of sorts, and no reason to act.
Note that cognition comes with no goals attached. Emotions is where the drive to do anything comes from. You allude to it in “contemplation of a possible choice or outcome”. My suspicion is that having a component not available to accurate introspection is essential to functioning. An agent (well, an algorithm) that can completely model itself (or, rather, model its identical copy, to mitigate the usual decision theory paradoxes), may end up in a state where there is no intrinsic meaning to anything, a nirvana of sorts, and no reason to act.