(2) We can try to build AIs that are not in this category, but screw up*
...
*(Remember, any AI is running searches through some space in pursuit of something, otherwise you would never call it “intelligence”. So one can imagine that the intelligent search may accidentally get aimed at the wrong target.)
The map is not the territory. A system can select a promising action from the space of possible actions without actually taking it. That said, there could be a risk of a “daemon” forming somehow.
I think I agree with this. The system is dangerous if its real-world output (pixels lit up on a display, etc.) is optimized to achieve a future-world-state. I guess that’s what I meant. If there are layers of processing that sit between the optimization process output and the real-world output, that seems like very much a step in the right direction. I dunno the details, it merits further thought.
The map is not the territory. A system can select a promising action from the space of possible actions without actually taking it. That said, there could be a risk of a “daemon” forming somehow.
I think I agree with this. The system is dangerous if its real-world output (pixels lit up on a display, etc.) is optimized to achieve a future-world-state. I guess that’s what I meant. If there are layers of processing that sit between the optimization process output and the real-world output, that seems like very much a step in the right direction. I dunno the details, it merits further thought.