We’ll start by defining “as useful for X as Hugh,” and then we will informally say that a program is “as useful” as Hugh if it’s as useful for the tasks we care most about.
If a program is useful accomplishing the tasks we care most about, while being horrible for the things we care less about, would the program still be considered useful? For example, suppose I care a lot about music, and just a little about comedy. If an AI was useful for making the music I listen to slightly better, but completely destroyed my ability to get comedy, I’m not sure it’s a good idea to call such a thing “useful”.
If a program is useful accomplishing the tasks we care most about, while being horrible for the things we care less about, would the program still be considered useful? For example, suppose I care a lot about music, and just a little about comedy. If an AI was useful for making the music I listen to slightly better, but completely destroyed my ability to get comedy, I’m not sure it’s a good idea to call such a thing “useful”.