One of the things I really like about this series is the way in which cognition is viewed as this double-edged sword, where it is specifically the things that make it good that also make it bad. The ability to quickly reach conclusions is both what makes intelligence useful—you need less sensory data / less time to decide things—and what makes it problematic—you jump to incorrect conclusions more quickly as well. This is, of course, also my view on AI alignment: the problem is not that people build robots and then foolishly decide to put guns on the robots. The problem is that we only know how to make the first-order cognition, where we know how to make optimizers that search across a wide possibility space for things that maximize some score, with no attention on whether or not they have the right score function. So the robots we build now are very susceptible to illusion and self-deception.
This also feels very tied to the spirit behind Less Wrong: intelligence and rationality are distinct things, where rationality is mostly focusing on the ways in which you personally are subject to illusion and self-deception, and need to rearrange your thinking such that your intelligence is helping you instead of an obstacle.
One of the things I really like about this series is the way in which cognition is viewed as this double-edged sword, where it is specifically the things that make it good that also make it bad. The ability to quickly reach conclusions is both what makes intelligence useful—you need less sensory data / less time to decide things—and what makes it problematic—you jump to incorrect conclusions more quickly as well. This is, of course, also my view on AI alignment: the problem is not that people build robots and then foolishly decide to put guns on the robots. The problem is that we only know how to make the first-order cognition, where we know how to make optimizers that search across a wide possibility space for things that maximize some score, with no attention on whether or not they have the right score function. So the robots we build now are very susceptible to illusion and self-deception.
This also feels very tied to the spirit behind Less Wrong: intelligence and rationality are distinct things, where rationality is mostly focusing on the ways in which you personally are subject to illusion and self-deception, and need to rearrange your thinking such that your intelligence is helping you instead of an obstacle.