The wish on top of that page is actually very problematic...
Yes, that was a part of the point of the article—people try to fully specify what they want, it gets this complex, and it’s still missing things; meanwhile, people understand what someone means when they say “I wish I was immortal.”
Right—there’s no misunderstanding, because the complexity is hidden by expectations and all sorts of shared stuff that isn’t likely to be there when talking to a genie of the “sufficiently sophisticated AI” variety, unless you are very careful about making sure that it is. Hence, the wish has hidden complexity—the point (and title) of the article.
Yes, that was a part of the point of the article—people try to fully specify what they want, it gets this complex, and it’s still missing things; meanwhile, people understand what someone means when they say “I wish I was immortal.”
Well, they understand it about as well as the speaker does. It’s not clear to me that the speaker always knows what they mean.
Right—there’s no misunderstanding, because the complexity is hidden by expectations and all sorts of shared stuff that isn’t likely to be there when talking to a genie of the “sufficiently sophisticated AI” variety, unless you are very careful about making sure that it is. Hence, the wish has hidden complexity—the point (and title) of the article.