Humans do not optimize things. We stack satisficer on satisficer and sometimes approximate optimization. That’s fine for us. But an AI would be working in a much larger mind space. It has no already established satisficing criteria. Deciding which satisficing criteria we want to use and which we do not is an enormously difficult problem.
Humans do not optimize things. We stack satisficer on satisficer and sometimes approximate optimization. That’s fine for us. But an AI would be working in a much larger mind space. It has no already established satisficing criteria. Deciding which satisficing criteria we want to use and which we do not is an enormously difficult problem.