Eh. I genuinely don’t expect to build an AI that acts like a utility maximizer in all contexts. All real-world agents are limited—they can get hit by radiation, or dumped into supernovae, or fed adversarial noise, etc. All we’re ever going to see in the real world are things that can be intentional-stanced in broad but limited domains.
Satisficers have goal-directed behavior sometimes, but not in all contexts—the more satisfied they are, the less goal-directed they are. If I built a satisficer who would be satisfied with merely controlling the Milky Way (rather than the entire universe), that’s plenty dangerous. And coincidentally, it’s going to be acting goal-directed in all contexts present in your everyday life, because none of them come close to satisfying it.
Eh. I genuinely don’t expect to build an AI that acts like a utility maximizer in all contexts. All real-world agents are limited—they can get hit by radiation, or dumped into supernovae, or fed adversarial noise, etc. All we’re ever going to see in the real world are things that can be intentional-stanced in broad but limited domains.
Satisficers have goal-directed behavior sometimes, but not in all contexts—the more satisfied they are, the less goal-directed they are. If I built a satisficer who would be satisfied with merely controlling the Milky Way (rather than the entire universe), that’s plenty dangerous. And coincidentally, it’s going to be acting goal-directed in all contexts present in your everyday life, because none of them come close to satisfying it.