This was a really good post, and I think accurately reflects a lot of people’s viewpoints. Thanks!
Working in finance, you find a lot of unnecessary jargon designed to keep smart laymen out of the discussion.
Most fields, especially technical fields, don’t do this. They use jargon because 1) the actual meanings the jargon points to don’t have short, precise, natural language equivalents, and 2) if experts did assign such short handles using normal language, the words and phrases used would still be prone to misunderstanding by non-experts because there are wide variations in non-technical usage, plus it would be harder for experts to know when their peers are speaking precisely vs. colloquially. In my own work, I will often be asked a question that I can figure out the overall answer to in 5 minutes, and I can express the answer and how I found it to my colleagues in seconds, but demonstrating it to others regularly takes over a day of effort organizing thoughts and background data and assumptions, and minutes to hours presenting and discussing it. I’m hardly the world’s best explainer, but this is a core part of my job for the past 12 years and I get lots of feedback indicating I’m pretty good at it.
We can argue over the hidden complexity of wishes, but
I think this section greatly underestimates just how much hidden complexity (EY and other high-probability-of-doom-predictors say that) wishes have. It’s not so much, “a longer sentence with more caveats would have been fine,” but rather more like “the required complexity has never been able to be even close to achieved or precisely described in all the verbal musings and written explorations of axiology/morality/ethics/law/politics/theology/psychology that humanity has ever produced since the dawn of language.” That claim may well be wrong, but it’s not a small difference of opinion.
If we are talking about an opaque black box, how can you be >90% confident about what it contains?
This is a disagreement over priors, not black boxes. I am much more than 90% certain that the interior of a black hole beyond the event horizon does not consist of a habitable environment full of happy, immortal, well-cared for puppies eternally enjoying themselves. I am also much more than 90% certain that if I plop a lump of graphite in water and seal it a time capsule for 30 years, that when I open it, it won’t contain diamonds and neatly-separated regions of hydrogen and oxygen gas. I’m not claiming anyone has that level of certainty of priors regarding AI x-risk, or even close. But if most possible good outcomes require complex specifications, that means there are orders of magnitude more ways for things to go wrong, than right. That’s a high bar for what level of caution and control is needed to steer towards good outcomes. Maybe not high enough to get to >90%, but high enough that I’d find it hard to be convinced of <10%. And my bar for saying “sure, let’s roll the dice on the entire future light cone of Earth” is way less than 10%.
This was a really good post, and I think accurately reflects a lot of people’s viewpoints. Thanks!
Most fields, especially technical fields, don’t do this. They use jargon because 1) the actual meanings the jargon points to don’t have short, precise, natural language equivalents, and 2) if experts did assign such short handles using normal language, the words and phrases used would still be prone to misunderstanding by non-experts because there are wide variations in non-technical usage, plus it would be harder for experts to know when their peers are speaking precisely vs. colloquially. In my own work, I will often be asked a question that I can figure out the overall answer to in 5 minutes, and I can express the answer and how I found it to my colleagues in seconds, but demonstrating it to others regularly takes over a day of effort organizing thoughts and background data and assumptions, and minutes to hours presenting and discussing it. I’m hardly the world’s best explainer, but this is a core part of my job for the past 12 years and I get lots of feedback indicating I’m pretty good at it.
I think this section greatly underestimates just how much hidden complexity (EY and other high-probability-of-doom-predictors say that) wishes have. It’s not so much, “a longer sentence with more caveats would have been fine,” but rather more like “the required complexity has never been able to be even close to achieved or precisely described in all the verbal musings and written explorations of axiology/morality/ethics/law/politics/theology/psychology that humanity has ever produced since the dawn of language.” That claim may well be wrong, but it’s not a small difference of opinion.
This is a disagreement over priors, not black boxes. I am much more than 90% certain that the interior of a black hole beyond the event horizon does not consist of a habitable environment full of happy, immortal, well-cared for puppies eternally enjoying themselves. I am also much more than 90% certain that if I plop a lump of graphite in water and seal it a time capsule for 30 years, that when I open it, it won’t contain diamonds and neatly-separated regions of hydrogen and oxygen gas. I’m not claiming anyone has that level of certainty of priors regarding AI x-risk, or even close. But if most possible good outcomes require complex specifications, that means there are orders of magnitude more ways for things to go wrong, than right. That’s a high bar for what level of caution and control is needed to steer towards good outcomes. Maybe not high enough to get to >90%, but high enough that I’d find it hard to be convinced of <10%. And my bar for saying “sure, let’s roll the dice on the entire future light cone of Earth” is way less than 10%.