Well, I too expect some form of something that we would call “AI”, before 2100. I can even buy into some form of accelerating progress, albeit the progress would be accelerating before the “AI” due to the tools using relevant technologies, and would not have that sharp of a break. I even do agree that there is a certain level of risk involved in all the future progress including progress of the software.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than “Steven Hawkins hooked up with FHI” or what ever you are outlining here.
You do frequently lament that the AI risk is underfunded, under-supported, and there’s under-awareness about it. In the hypothetical world, this is not the case and you can only lament that the rational spending should be 2 billions rather than 1 billion.
edit: and of course, my true rejection is that I do not actually see rational inferences leading there. The imaginary world stuff is just a side-note to explain how non-experts generally look at it.
edit2: and I have nothing against FHI’s existence and their work. I don’t think they are very useful, or address any actual safety issues which may arise, though, but with them I am fairly certain they aren’t doing any harm either (Or at least, the possible harm would be very small). Promoting the idea that AI is possible within 100 years, however, is something that increases funding for AI all across the board.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than “Steven Hawkins hooked up with FHI” or what ever you are outlining here.
Right, this just goes back to the same disagreement in our models I was trying to address earlier by making predictions. Let me try something else, then. Here are some relevant parts of my model:
I expect most highly credentialed people to not be EAs in the first place.
I expect most highly credential people to be mostly just aware of risks they happen to have heard about (e.g. climate change, asteroids, nuclear war), rather than attempting a systematic review of risks (e.g. by reading the GCR volume).
I expect most highly credentialed people to respond fairly well when actuarial risk is easily calculated (e.g. asteroid risk), and not-so-well when it’s more difficult to calculate (e.g. many insurance companies went bankrupt after 9/11).
I expect most highly credentialed people to have spent little time on explicit calibration training.
I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
I expect most highly credentialed people to know very little about AI, and very little about AI risk.
I expect that in general, even those highly credentialed people who intuitively think AI risk is a big deal will not even contact the people who think about AI risk for a living in order to ask about their views and their reasons for them, due to basic VoI failure.
I expect most highly credentialed people to have fairly reasonable views within their own field, but to often have crazy views “outside the laboratory.”
I expect most highly credentialed people to not have a good understanding of Bayesian epistemology.
I expect most highly credentialed people to continue working on, and caring about, whatever their career has been up to that point, rather than suddenly switching career paths on the basis of new information and an EV calculation.
I expect most highly credentialed people to not understand lots of pieces of “black swan epistemology” like this one and this one.
The question should not be about “highly credentialed” people alone, but about how they fare compared to people who are rather very low “credentialed”.
In particular, on your list, I expect people with fairly low credentials to fare much worse, especially at identification of the important issues as well as on rational thinking. Those combine multiplicatively, making it exceedingly unlikely—despite the greater numbers of the credential-less masses—that people who lead the work on an important issue would have low credentials.
I expect most highly credentialed people to not be EAs in the first place.
What’s EA? Effective altruism? If it’s an existential risk, it kills everyone, selfishness suffices just fine.
e.g. many insurance companies went bankrupt after 9/11
Ohh, come on. That is in no way a demonstration that insurance companies in general follow faulty strategies, and especially is not a demonstration that you could do better.
I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
In particular, on your list, I expect people with fairly low credentials to fare much worse
No doubt! I wasn’t comparing highly credentialed people to low-credentialed people in general. I was comparing highly credentialed people to Bostrom, Yudkowsky, Shulman, etc.
But why exactly would you expect conventional researchers in AI and related technologies (also including provable software, as used in the aerospace industry, and a bunch of other topics), with credentials and/or accomplishments in said fields, to fare worse on that list’s score?
Furthermore, with regards to the rationality, risks of mistake, and such… very little was done that can be checked for correctness in a clear cut way—most is of such nature that even when wrong it would not be possible to conclusively demonstrate it wrong. The few things that can be checked… look, when you write an article like this , discussing irrationality of Enrico Fermi, there’s a substantial risk of appearing highly arrogant (and irrational) if you get the technical details wrong. It is a miniature version of AI risk problem—you need to understand the subject, and if you don’t, there’s negative consequences. It is much, much easier to not goof up in things like that, than AI direction.
As you guys are researching into actual AI technologies, the issue is that one should be able to deem your effort less of a risk. Mere “we are trying to avoid risk and we think they don’t” can’t do. The cost of a particularly bad friendly AI goof-up is a sadistic AI (to borrow the term from Omohundro). A sadistic AI can probably run far more tortured minds than a friendly AI can run minds, by a very huge factor, so the risk of a goof up must be quite a lot lower than anyone demonstrated.
BTW, I went back and numbered the items in my list so they’re easier to refer to.
But why exactly would you expect conventional researchers in AI and related technologies… with credentials and/or accomplishments in said fields, to fare worse on that list’s score?
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7)†, (8), (10), and (12), but Bostrom, Yudkowsky and Shulman rather uncontroversially do satisfy those items. I also expect B/Y/S to outperform most credentialed experts on (4), (9), and (11), but I understand that’s a subjective judgment call and it would take a long time for me to communicate my reasons.
† The AI risk part of 7, anyway. Obviously, AI people specifically know a lot about AI.
Edit: Also, I’ll briefly mention that I haven’t downvoted any of your comments in this conversation.
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7), (8), (10), and (12)
Ok, let’s go over your list, for the AI people.
1 I expect most highly credentialed people to not be EAs in the first place.
If EA is effective altruism, that’s not relevant because one doesn’t have to be an altruist to care about existential risks.
2 I expect most highly credentialed people to not be familiar with the arguments for caring about the far future.
I expect them to be able to come up with that independently if it is a good idea.
3 I expect most highly credential people to be mostly just aware of risks they happen to have heard about (e.g. climate change, asteroids, nuclear war), rather than attempting a systematic review of risks (e.g. by reading the GCR volume).
I expect intelligent people to be able to foresee risks, especially when prompted by the cultural baggage (modern variations on the theme of Golem)
4 I expect most highly credentialed people to respond fairly well when actuarial risk is easily calculated (e.g. asteroid risk), and not-so-well when it’s more difficult to calculate (e.g. many insurance companies went bankrupt after 9/11).
Well, that ought to imply some generally better ability to evaluate hard to calculate probabilities, which would imply that you guys should be able to make quite a bit of money.
5 I expect most highly credentialed people to have spent little time on explicit calibration training.
The question is how well are they calibrated, not how much time they spent. You guys see miscalibration of famous people everywhere, even in Enrico Fermi.
6 I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
Once again, how unbiased is what’s important, not how much time spent on a very specific way to acquire an ability. I expect most accomplished people to have encountered far more feedback on being right / being wrong through their education and experience.
7 I expect most highly credentialed people to know very little about AI, and very little about AI risk.
Doesn’t apply to people in AI related professions.
8 I expect that in general, even those highly credentialed people who intuitively think AI risk is a big deal will not even contact the people who think about AI risk for a living in order to ask about their views and their reasons for them, due to basic VoI failure.
The way to raise VoI is prior history of thinking about something else for a living, with impressive results.
9 I expect most highly credentialed people to have fairly reasonable views within their own field, but to often have crazy views “outside the laboratory.”
Well, less credentialed people are just like this except they don’t have a laboratory inside of which they are sane, that’s usually why they are less credentialed in the first place.
10 I expect most highly credentialed people to not have a good understanding of Bayesian epistemology.
Of your 3, I only weakly expect Bostrom to have learned the necessary fundamentals for actually applying Bayes theorem correctly in somewhat non-straightforward cases.
Yes, the basic formula is simple, but derivations are subtle and complex for non independent evidence or cases involving loops in the graph or all those other things…
It’s like arguing that you are better equipped for a job at Weta Digital than any employee there because you know quantum electrodynamics (the fundamentals of light propagation), and they’re using geometrical optics.
I expect many AI researchers to understand the relevant mathematics a lot, lot better than the 3 on your list.
And I expect credentialed people in general to have a good understanding of the variety of derivative tricks that are used to obtain effective results under uncertainty when the Bayes theorem can not be effectively applied.
11 I expect most highly credentialed people to continue working on, and caring about, whatever their career has been up to that point, rather than suddenly switching career paths on the basis of new information and an EV calculation.
Yeah, well, and I expect non-credentialed people to have too much to lose from backing out of it in the event that the studies return a negative.
12 I expect most highly credentialed people to not understand lots of pieces of “black swan epistemology” like this one and this one.
You lose me here.
I would make a different list, anyway. There’s my list:
Relevant expertise as measured by educational credentials and/or accomplishments. Expertise is required for correctly recognizing risks (e.g. an astronomer is better equipped for recognizing risks from the outer space, a physicist for recognizing faults in a nuclear power plant design, et cetera)
Proven ability to make correct inferences (largely required for 1).
Self preservation (most of us have it)
Lack of 1 is an automatic dis-qualifier in my list. It doesn’t matter how much you are into things that you think are important for identifying, say, faults in a nuclear power plant design. If you are not an engineer, a physicist, or the like, you aren’t going to qualify for that job via some list you make yourself, which conveniently omits (1).
I disagree with many of your points, but I don’t have time to reply to all that, so to avoid being logically rude I’ll at least reply to what seems to be your central point, about “relevant expertise as measured by educational credentials and/or accomplishments.”
Who has educational credentials and/or accomplishments relevant to future AGI designs or long-term tech forecasting? Also, do you particularly disagree with what I wrote in AGI Impact Experts and Friendly AI Experts?
Also, in general, I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts (e.g. facts relevant to the theses in this post). Argument screens off authority, and all that.
Edit: Also, my view of Holden Karnofsky might be illustrative. I take Holden Karnofsky more seriously than almost anyone on the cost-effectiveness of global health interventions, despite the fact that he has 0 relevant degrees, 0 papers published in relevant journals, 0 awards for global health work, etc. Degrees and papers and so on are only proxy variables for what we really care about, and are easily screened off by more relevant variables, both for the case of Karnofsky on global health and for the case of Bostrom, Yudkowsky, Shulman, etc. on AI risk.
For Karnofsky and to some extent Bostrom yes, Shulman is debatable, Yudkowsky tried to get screened (tried to write a programming language, for example, wrote a lot of articles on various topics, many of them wrong, tried to write technical papers (TDT), really badly), and failed to pass the screening by a very big margin. Entirely irrational arguments about 10% counter-factual impact of his are also a part of failure. Omohundro passed with flying colours (his PhD is almost entirely irrelevant at that point, as it is screened off by his accomplishments in AI).
I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts....
Exactly. All of this is wasted effort once either FAI or UFAI is developed.
Who has educational credentials and/or accomplishments relevant to future AGI designs or long-term tech forecasting?
There’s the more relevant accomplishments, there are less relevant accomplishments, and lacks of accomplishment.
Also, in general, I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts
I agree that a discussion of strategically relevant facts would be much more productive. I don’t see facts here. I see many speculations. I see a lot of making things up to fit the conclusion.
If I were to tell you that I can, for example, win a very high stakes programming contest (with a difficult, open problem that has many potential solutions that can be ranked in terms of quality), the discussion of my approach to the contest problem between you and me would be almost useless for your or my prediction of victory (provided that basic standards of competence are met), irrespective of whenever my idea is good. Prior track record, on the other hand, would be a good predictor. This is how it is for a very well defined problem. It is not going to be better for a less well understood problem.
If EA is effective altruism, that’s not relevant because one doesn’t have to be an altruist to care about existential risks.
‘EA’ here refers to the traits a specific community seems to exemplify (though those traits may occur outside the community). So more may be suggested than the words ‘effective’ and ‘altruism’ contain.
In terms of the terms, I think ‘altruism’ here is supposed to be an inclination to behave a certain way, not an other-privileging taste or ideology. Think ‘reciprocal altruism’. You can be an egoist who’s an EA, provided your selfish calculation has led you to the conclusion that you should devote yourself to efficiently funneling money to the world’s poorest, efficiently reducing existential risks, etc. I’m guessing by ‘EA’ Luke has in mind a set of habits of looking at existential risks that ‘Effective Altruists’ tend to exemplify, e.g., quantifying uncertainty, quantifying benefit, strongly attending to quantitative differences, trying strongly to correct for a specific set of biases (absurdity bias, status quo bias, optimism biases, availability biases), relying heavily on published evidence, scrutinizing the methodology and interpretation of published evidence....
I expect them to be able to come up with that independently if it is a good idea.
My own experience is that I independently came up with a lot of arguments from the Sequences, but didn’t take them sufficiently seriously, push them hard enough, or examine them in enough detail. There seems to be a big gap between coming up with an abstract argument for something while you’re humming in the shower, and actually living your life in a way that’s consistent with your believing the argument is sound.
My own experience is that I independently came up with a lot of arguments from the Sequences, but didn’t take them sufficiently seriously, push them hard enough, or examine them in enough detail.
But we are speaking of credentialed people. They’re fairly driven.
Furthermore, general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views. (The biggest problem with “Bayesianism” is dis-balanced/selective updates)
First off, if one can support existential risk for non Pascal’s wager type reasons then enormous utility of the future should not be relevant. If it is actually a requirement then I don’t think there’s anything to discuss here.
Secondarily, the most common norm of morality (Assuming we ignore things like Sharia), as specified in the laws of progressive countries, or as extrapolation of legal progress in less progressive ones, is to value the future people (we disapprove of smoking while pregnant), but not value counter-factual creation of future people (we allow abortion, and especially when the child would be disadvantaged and not have a fair chance). Rather than inferring the prevailing morality from the law and discussing it, various bad ideas are invented and discussed to make the argument appear stronger than it really is.
It is not that I am not exposed to this worldview. I am. It is that choosing between A: hurt someone, but a large number of happy people will be created, and B: not hurt someone, but a large number of happy people will not be created (with the deliberate choice having the causal impact on the hurting and creation), A is both illegal and immoral.
general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views.
When I hear that Joe has a new argument against a belief of mine, then my confidence in my belief lowers a bit, and my confidence in Joe’s competence also lowers a bit. If I then go on to actually evaluate the argument in detail and discover that it’s an extraordinarily poor one, this should generally increase my confidence to higher than it was before I heard that Joe had an argument, and it should further lower my confidence in Joe’s competence.
I’ve spent enough time looking at the specific arguments for and against many of these propositions to have the contents of those arguments overwhelm my expertise priors in both directions, such that I just don’t see a whole lot of value in discussing anything but the arguments themselves, when my goal (and yours) is to figure out the level of merit of the arguments.
if one can support existential risk for non Pascal’s wager type reasons then enormous utility of the future should not be relevant.
It sounds like you’re committing the Pascal’s Wager Fallacy Fallacy. If you aren’t, then I’m not understanding your point. Large future utilities should count more than small future utilities, and multiplying by low probabilities is fine if the probabilities aren’t vanishingly low.
Choosing between A: hurt someone, but a large number of happy people get created, and B: not hurt someone, but a large number of happy people do not get created, A is both illegal and immoral.
I think there’s a quantitative tradeoff between the happiness of currently existent people and the happiness of possibly-created people. A strict rule ‘Counterfactual People Have Absolutely No Value’ leads to absurd conclusions, e.g., it’s not worthwhile to create an infinite number of infinitely happy and well-off people if the cost is that your shoulder itches for a few seconds. It’s at least a little worthwhile to create people with awesome lives, even if they should get weighted less than currently existent people.
I’ve spent enough time looking at the specific arguments for and against many of these propositions to have the contents of those arguments overwhelm my expertise priors in both directions, such that I just don’t see a whole lot of value in discussing anything but the arguments themselves, when my goal (and yours) is to figure out the level of merit of the arguments.
You don’t want the outcome to be biased by the availability of the arguments, right? Really, I think you do not account for the fact that the available arguments are merely samples from the space of possible arguments (which make different speculative assumptions, in a very large space of possible speculations). Picked non uniformly, too, as arguments for one side may be more available, or their creation may maximize personal present-day utility of more agents. Individual samples can’t be particularly informative in such a situation.
It’s at least a little worthwhile to create people with awesome lives, even if they should get weighted less than currently existent people.
The issue is that the number of people you can speculate you affect grows much faster than the prior for the speculation decreases. Constant factors do not help with that, they just push the problem a little further.
A strict rule ‘Counterfactual People Have Absolutely No Value’ leads to absurd conclusions, e.g., it’s not worthwhile to create an infinite number of infinitely happy and well-off people if the cost is that your shoulder itches for a few seconds.
I don’t see that as problematic. Ponder the alternative for a moment: you may be ok with a shoulder itch, but are you OK with 10 000 years of the absolutely worst torture imaginable, for the sake of creation of 3^^^3 or 3^^^^^3 or however many really happy people? What’s about your death vs their creation?
edit: also you might have the value of those people to yourself (as potential mates and whatnot) leaking in.
It sounds like you’re committing the Pascal’s Wager Fallacy Fallacy. If you aren’t, then I’m not understanding your point. Large future utilities should count more than small future utilities, and multiplying by low probabilities is fine if the probabilities aren’t vanishingly low.
If the probabilities aren’t vanishingly low, you reach basically same conclusions without requiring extremely large utilities. 7 billion people dying is quite a lot, too. If you see extremely large utilities on a list of requirements for caring about the issue, when you already have at least 7 billion lives at stake, then it is a Pascal’s wager.
Actually, I don’t see vanishingly small probabilities problematic, I see small probabilities where the bulk of probability mass is unaccounted for, problematic. E.g. response to low risk from a specific asteroid is fine, because it’s alternative positions in space are accounted for (and you have assurance you won’t put it on an even worse trajectory)
Furthermore, general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views. (The biggest problem with “Bayesianism” is dis-balanced/selective updates)
Updating on someone else’s decision to accept or reject a position should depend on their reason for their position. Information cascades is relevant.
Yes, of course. But also keep in mind that wrong positions are often rejected by the mechanism that generates positions, rather than the mechanism that checks the generated positions.
Well, I too expect some form of something that we would call “AI”, before 2100. I can even buy into some form of accelerating progress, albeit the progress would be accelerating before the “AI” due to the tools using relevant technologies, and would not have that sharp of a break. I even do agree that there is a certain level of risk involved in all the future progress including progress of the software.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than “Steven Hawkins hooked up with FHI” or what ever you are outlining here.
You do frequently lament that the AI risk is underfunded, under-supported, and there’s under-awareness about it. In the hypothetical world, this is not the case and you can only lament that the rational spending should be 2 billions rather than 1 billion.
edit: and of course, my true rejection is that I do not actually see rational inferences leading there. The imaginary world stuff is just a side-note to explain how non-experts generally look at it.
edit2: and I have nothing against FHI’s existence and their work. I don’t think they are very useful, or address any actual safety issues which may arise, though, but with them I am fairly certain they aren’t doing any harm either (Or at least, the possible harm would be very small). Promoting the idea that AI is possible within 100 years, however, is something that increases funding for AI all across the board.
Right, this just goes back to the same disagreement in our models I was trying to address earlier by making predictions. Let me try something else, then. Here are some relevant parts of my model:
I expect most highly credentialed people to not be EAs in the first place.
I expect most highly credentialed people to not be familiar with the arguments for caring about the far future.
I expect most highly credential people to be mostly just aware of risks they happen to have heard about (e.g. climate change, asteroids, nuclear war), rather than attempting a systematic review of risks (e.g. by reading the GCR volume).
I expect most highly credentialed people to respond fairly well when actuarial risk is easily calculated (e.g. asteroid risk), and not-so-well when it’s more difficult to calculate (e.g. many insurance companies went bankrupt after 9/11).
I expect most highly credentialed people to have spent little time on explicit calibration training.
I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
I expect most highly credentialed people to know very little about AI, and very little about AI risk.
I expect that in general, even those highly credentialed people who intuitively think AI risk is a big deal will not even contact the people who think about AI risk for a living in order to ask about their views and their reasons for them, due to basic VoI failure.
I expect most highly credentialed people to have fairly reasonable views within their own field, but to often have crazy views “outside the laboratory.”
I expect most highly credentialed people to not have a good understanding of Bayesian epistemology.
I expect most highly credentialed people to continue working on, and caring about, whatever their career has been up to that point, rather than suddenly switching career paths on the basis of new information and an EV calculation.
I expect most highly credentialed people to not understand lots of pieces of “black swan epistemology” like this one and this one.
etc.
Luke, why are you arguing with Dmytry?
The question should not be about “highly credentialed” people alone, but about how they fare compared to people who are rather very low “credentialed”.
In particular, on your list, I expect people with fairly low credentials to fare much worse, especially at identification of the important issues as well as on rational thinking. Those combine multiplicatively, making it exceedingly unlikely—despite the greater numbers of the credential-less masses—that people who lead the work on an important issue would have low credentials.
What’s EA? Effective altruism? If it’s an existential risk, it kills everyone, selfishness suffices just fine.
Ohh, come on. That is in no way a demonstration that insurance companies in general follow faulty strategies, and especially is not a demonstration that you could do better.
Indeed.
A selfish person protecting against existential risk builds a bunker and stocks it with sixty years of foodstuffs. That doesn’t exactly help much.
For what existential risks is this actually an effective strategy?
A global pandemic that kills everyone?
The quality of life in a bunker is really damn low. Not to mention that you presumably won’t survive this particular risk in a bunker.
No doubt! I wasn’t comparing highly credentialed people to low-credentialed people in general. I was comparing highly credentialed people to Bostrom, Yudkowsky, Shulman, etc.
But why exactly would you expect conventional researchers in AI and related technologies (also including provable software, as used in the aerospace industry, and a bunch of other topics), with credentials and/or accomplishments in said fields, to fare worse on that list’s score?
Furthermore, with regards to the rationality, risks of mistake, and such… very little was done that can be checked for correctness in a clear cut way—most is of such nature that even when wrong it would not be possible to conclusively demonstrate it wrong. The few things that can be checked… look, when you write an article like this , discussing irrationality of Enrico Fermi, there’s a substantial risk of appearing highly arrogant (and irrational) if you get the technical details wrong. It is a miniature version of AI risk problem—you need to understand the subject, and if you don’t, there’s negative consequences. It is much, much easier to not goof up in things like that, than AI direction.
As you guys are researching into actual AI technologies, the issue is that one should be able to deem your effort less of a risk. Mere “we are trying to avoid risk and we think they don’t” can’t do. The cost of a particularly bad friendly AI goof-up is a sadistic AI (to borrow the term from Omohundro). A sadistic AI can probably run far more tortured minds than a friendly AI can run minds, by a very huge factor, so the risk of a goof up must be quite a lot lower than anyone demonstrated.
BTW, I went back and numbered the items in my list so they’re easier to refer to.
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7)†, (8), (10), and (12), but Bostrom, Yudkowsky and Shulman rather uncontroversially do satisfy those items. I also expect B/Y/S to outperform most credentialed experts on (4), (9), and (11), but I understand that’s a subjective judgment call and it would take a long time for me to communicate my reasons.
† The AI risk part of 7, anyway. Obviously, AI people specifically know a lot about AI.
Edit: Also, I’ll briefly mention that I haven’t downvoted any of your comments in this conversation.
Ok, let’s go over your list, for the AI people.
If EA is effective altruism, that’s not relevant because one doesn’t have to be an altruist to care about existential risks.
I expect them to be able to come up with that independently if it is a good idea.
I expect intelligent people to be able to foresee risks, especially when prompted by the cultural baggage (modern variations on the theme of Golem)
Well, that ought to imply some generally better ability to evaluate hard to calculate probabilities, which would imply that you guys should be able to make quite a bit of money.
The question is how well are they calibrated, not how much time they spent. You guys see miscalibration of famous people everywhere, even in Enrico Fermi.
Once again, how unbiased is what’s important, not how much time spent on a very specific way to acquire an ability. I expect most accomplished people to have encountered far more feedback on being right / being wrong through their education and experience.
Doesn’t apply to people in AI related professions.
The way to raise VoI is prior history of thinking about something else for a living, with impressive results.
Well, less credentialed people are just like this except they don’t have a laboratory inside of which they are sane, that’s usually why they are less credentialed in the first place.
Of your 3, I only weakly expect Bostrom to have learned the necessary fundamentals for actually applying Bayes theorem correctly in somewhat non-straightforward cases.
Yes, the basic formula is simple, but derivations are subtle and complex for non independent evidence or cases involving loops in the graph or all those other things…
It’s like arguing that you are better equipped for a job at Weta Digital than any employee there because you know quantum electrodynamics (the fundamentals of light propagation), and they’re using geometrical optics.
I expect many AI researchers to understand the relevant mathematics a lot, lot better than the 3 on your list.
And I expect credentialed people in general to have a good understanding of the variety of derivative tricks that are used to obtain effective results under uncertainty when the Bayes theorem can not be effectively applied.
Yeah, well, and I expect non-credentialed people to have too much to lose from backing out of it in the event that the studies return a negative.
You lose me here.
I would make a different list, anyway. There’s my list:
Relevant expertise as measured by educational credentials and/or accomplishments. Expertise is required for correctly recognizing risks (e.g. an astronomer is better equipped for recognizing risks from the outer space, a physicist for recognizing faults in a nuclear power plant design, et cetera)
Proven ability to make correct inferences (largely required for 1).
Self preservation (most of us have it)
Lack of 1 is an automatic dis-qualifier in my list. It doesn’t matter how much you are into things that you think are important for identifying, say, faults in a nuclear power plant design. If you are not an engineer, a physicist, or the like, you aren’t going to qualify for that job via some list you make yourself, which conveniently omits (1).
edit: list copy paste failed.
I disagree with many of your points, but I don’t have time to reply to all that, so to avoid being logically rude I’ll at least reply to what seems to be your central point, about “relevant expertise as measured by educational credentials and/or accomplishments.”
Who has educational credentials and/or accomplishments relevant to future AGI designs or long-term tech forecasting? Also, do you particularly disagree with what I wrote in AGI Impact Experts and Friendly AI Experts?
Also, in general, I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts (e.g. facts relevant to the theses in this post). Argument screens off authority, and all that.
Edit: Also, my view of Holden Karnofsky might be illustrative. I take Holden Karnofsky more seriously than almost anyone on the cost-effectiveness of global health interventions, despite the fact that he has 0 relevant degrees, 0 papers published in relevant journals, 0 awards for global health work, etc. Degrees and papers and so on are only proxy variables for what we really care about, and are easily screened off by more relevant variables, both for the case of Karnofsky on global health and for the case of Bostrom, Yudkowsky, Shulman, etc. on AI risk.
For Karnofsky and to some extent Bostrom yes, Shulman is debatable, Yudkowsky tried to get screened (tried to write a programming language, for example, wrote a lot of articles on various topics, many of them wrong, tried to write technical papers (TDT), really badly), and failed to pass the screening by a very big margin. Entirely irrational arguments about 10% counter-factual impact of his are also a part of failure. Omohundro passed with flying colours (his PhD is almost entirely irrelevant at that point, as it is screened off by his accomplishments in AI).
Exactly. All of this is wasted effort once either FAI or UFAI is developed.
There’s the more relevant accomplishments, there are less relevant accomplishments, and lacks of accomplishment.
I agree that a discussion of strategically relevant facts would be much more productive. I don’t see facts here. I see many speculations. I see a lot of making things up to fit the conclusion.
If I were to tell you that I can, for example, win a very high stakes programming contest (with a difficult, open problem that has many potential solutions that can be ranked in terms of quality), the discussion of my approach to the contest problem between you and me would be almost useless for your or my prediction of victory (provided that basic standards of competence are met), irrespective of whenever my idea is good. Prior track record, on the other hand, would be a good predictor. This is how it is for a very well defined problem. It is not going to be better for a less well understood problem.
‘EA’ here refers to the traits a specific community seems to exemplify (though those traits may occur outside the community). So more may be suggested than the words ‘effective’ and ‘altruism’ contain.
In terms of the terms, I think ‘altruism’ here is supposed to be an inclination to behave a certain way, not an other-privileging taste or ideology. Think ‘reciprocal altruism’. You can be an egoist who’s an EA, provided your selfish calculation has led you to the conclusion that you should devote yourself to efficiently funneling money to the world’s poorest, efficiently reducing existential risks, etc. I’m guessing by ‘EA’ Luke has in mind a set of habits of looking at existential risks that ‘Effective Altruists’ tend to exemplify, e.g., quantifying uncertainty, quantifying benefit, strongly attending to quantitative differences, trying strongly to correct for a specific set of biases (absurdity bias, status quo bias, optimism biases, availability biases), relying heavily on published evidence, scrutinizing the methodology and interpretation of published evidence....
My own experience is that I independently came up with a lot of arguments from the Sequences, but didn’t take them sufficiently seriously, push them hard enough, or examine them in enough detail. There seems to be a big gap between coming up with an abstract argument for something while you’re humming in the shower, and actually living your life in a way that’s consistent with your believing the argument is sound.
But we are speaking of credentialed people. They’re fairly driven.
Furthermore, general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views. (The biggest problem with “Bayesianism” is dis-balanced/selective updates)
In particular, when it comes to the interview that he linked for reasons why value the future…
First off, if one can support existential risk for non Pascal’s wager type reasons then enormous utility of the future should not be relevant. If it is actually a requirement then I don’t think there’s anything to discuss here.
Secondarily, the most common norm of morality (Assuming we ignore things like Sharia), as specified in the laws of progressive countries, or as extrapolation of legal progress in less progressive ones, is to value the future people (we disapprove of smoking while pregnant), but not value counter-factual creation of future people (we allow abortion, and especially when the child would be disadvantaged and not have a fair chance). Rather than inferring the prevailing morality from the law and discussing it, various bad ideas are invented and discussed to make the argument appear stronger than it really is.
It is not that I am not exposed to this worldview. I am. It is that choosing between A: hurt someone, but a large number of happy people will be created, and B: not hurt someone, but a large number of happy people will not be created (with the deliberate choice having the causal impact on the hurting and creation), A is both illegal and immoral.
When I hear that Joe has a new argument against a belief of mine, then my confidence in my belief lowers a bit, and my confidence in Joe’s competence also lowers a bit. If I then go on to actually evaluate the argument in detail and discover that it’s an extraordinarily poor one, this should generally increase my confidence to higher than it was before I heard that Joe had an argument, and it should further lower my confidence in Joe’s competence.
I’ve spent enough time looking at the specific arguments for and against many of these propositions to have the contents of those arguments overwhelm my expertise priors in both directions, such that I just don’t see a whole lot of value in discussing anything but the arguments themselves, when my goal (and yours) is to figure out the level of merit of the arguments.
It sounds like you’re committing the Pascal’s Wager Fallacy Fallacy. If you aren’t, then I’m not understanding your point. Large future utilities should count more than small future utilities, and multiplying by low probabilities is fine if the probabilities aren’t vanishingly low.
I think there’s a quantitative tradeoff between the happiness of currently existent people and the happiness of possibly-created people. A strict rule ‘Counterfactual People Have Absolutely No Value’ leads to absurd conclusions, e.g., it’s not worthwhile to create an infinite number of infinitely happy and well-off people if the cost is that your shoulder itches for a few seconds. It’s at least a little worthwhile to create people with awesome lives, even if they should get weighted less than currently existent people.
You don’t want the outcome to be biased by the availability of the arguments, right? Really, I think you do not account for the fact that the available arguments are merely samples from the space of possible arguments (which make different speculative assumptions, in a very large space of possible speculations). Picked non uniformly, too, as arguments for one side may be more available, or their creation may maximize personal present-day utility of more agents. Individual samples can’t be particularly informative in such a situation.
The issue is that the number of people you can speculate you affect grows much faster than the prior for the speculation decreases. Constant factors do not help with that, they just push the problem a little further.
I don’t see that as problematic. Ponder the alternative for a moment: you may be ok with a shoulder itch, but are you OK with 10 000 years of the absolutely worst torture imaginable, for the sake of creation of 3^^^3 or 3^^^^^3 or however many really happy people? What’s about your death vs their creation?
edit: also you might have the value of those people to yourself (as potential mates and whatnot) leaking in.
forgot to address this:
If the probabilities aren’t vanishingly low, you reach basically same conclusions without requiring extremely large utilities. 7 billion people dying is quite a lot, too. If you see extremely large utilities on a list of requirements for caring about the issue, when you already have at least 7 billion lives at stake, then it is a Pascal’s wager.
Actually, I don’t see vanishingly small probabilities problematic, I see small probabilities where the bulk of probability mass is unaccounted for, problematic. E.g. response to low risk from a specific asteroid is fine, because it’s alternative positions in space are accounted for (and you have assurance you won’t put it on an even worse trajectory)
Updating on someone else’s decision to accept or reject a position should depend on their reason for their position. Information cascades is relevant.
Yes, of course. But also keep in mind that wrong positions are often rejected by the mechanism that generates positions, rather than the mechanism that checks the generated positions.