wireheading that does not involve creating as many sentient organisms as possible
Sure. For instance, consider the directive “Make everyone on Earth live as happily as possible for as long as the sun lasts.” Solution: Wirehead everyone, then figure out how to blow up the sun, then shut down — mission accomplished.
Not if the system is optimizing for the probability of success and can cheaply send out probes to eat the universe and use it to make sure the job is finished lest something go wrong (e.g. the sun-destroyer [???] failed, or aliens resuscitate the Sun under whatever criterion of stellar life is used).
“Our analysis of the alien probe shows that its intended function is to … um … go back to a star that blew up ten thousand years ago and make damn sure that it’s blown up and not just fooling.”
Sure. For instance, consider the directive “Make everyone on Earth live as happily as possible for as long as the sun lasts.” Solution: Wirehead everyone, then figure out how to blow up the sun, then shut down — mission accomplished.
Not if the system is optimizing for the probability of success and can cheaply send out probes to eat the universe and use it to make sure the job is finished lest something go wrong (e.g. the sun-destroyer [???] failed, or aliens resuscitate the Sun under whatever criterion of stellar life is used).
“Our analysis of the alien probe shows that its intended function is to … um … go back to a star that blew up ten thousand years ago and make damn sure that it’s blown up and not just fooling.”