One big difference is that “having enough food” admits a value function (“quantity of food”) that is both well understood and for the most part smooth and continuous over the design space, given today’s design methodology (if we try to design a ship with a particular amount of food and make a tiny mistake it’s unlikely that the quantity of food will change that much). In contrast, the “how well is it aligned” metric is very poorly understood (at least compared with “amount of food on a spaceship”) and a lot more discontinuous (using today’s techniques of designing AIs, a tiny error in alignment is almost certain to cause catastrophic failure). Basically—we do not know what exactly if means to get it right, and even if we knew, we do not know what the acceptable error tolerances are, and even if we knew, we do not know how to meet them. None of that applies to the amount of food on a spaceship.
One big difference is that “having enough food” admits a value function (“quantity of food”) that is both well understood and for the most part smooth and continuous over the design space, given today’s design methodology (if we try to design a ship with a particular amount of food and make a tiny mistake it’s unlikely that the quantity of food will change that much). In contrast, the “how well is it aligned” metric is very poorly understood (at least compared with “amount of food on a spaceship”) and a lot more discontinuous (using today’s techniques of designing AIs, a tiny error in alignment is almost certain to cause catastrophic failure). Basically—we do not know what exactly if means to get it right, and even if we knew, we do not know what the acceptable error tolerances are, and even if we knew, we do not know how to meet them. None of that applies to the amount of food on a spaceship.