Unfortunately, we find ourselves in a world where the world’s policy-makers don’t just profess that AGI safety isn’t a pressing issue, they also aren’t taking any action on AGI safety. Even generally sharp people like Bryan Caplan give disappointingly lame reasons for not caring. :(
Why won’t you update towards the possibility that they’re right and you’re wrong?
This model should rise up much sooner than some very low prior complex model where you’re a better truth finder about this topic but not any topic where truth-finding can be tested reliably*, and they’re better truth finders about topics where truth finding can be tested (which is what happens when they do their work), but not this particular topic.
(*because if you expect that, then you should end up actually trying to do at least something that can be checked because it’s the only indicator that you might possibly be right about the matters that can’t be checked in any way)
Why are the updates always in one direction only? When they disagree, the reasons are “lame” according to yourself, which makes you more sure everyone’s wrong. When they agree, they agree and that makes you more sure you are right.
This model should rise up much sooner than some very low prior complex model where you’re a better truth finder about this topic...
It’s not so much that I’m a better truth finder, it’s that I’ve had the privilege of thinking through the issues as a core component of my full time job for the past two years, and people like Caplan only raise points that have been accounted for in my model for a long time. Also, I think the most productive way to resolve these debates is not to argue the meta-level issues about social epistemology, but to have the object-level debates about the facts at issue. So if Caplan replies to Carl’s comment and my own, then we can continue the object-level debate, otherwise… the ball’s in his court.
Why are the updates always in one direction only? When they disagree, the reasons are “lame” according to yourself, which makes you more sure everyone’s wrong. When they agree, they agree and that makes you more sure you are right.
This doesn’t appear to be accurate. E.g. Carl & Paul changed my mind about the probability of hard takeoff. And when have I said that some public figure agreeing with me made me more sure I’m right? See also my comments here.
If I mention a public figure agreeing with me, it’s generally not because this plays a significant role in my own estimates, it’s because other people think there’s a stronger correlation between social status and correctness than I do.
It’s not so much that I’m a better truth finder, it’s that I’ve had the privilege of thinking through the issues as a core component of my full time job for the past two years, and people like Caplan only raise points that have been accounted for in my model for a long time.
Yes, but why Caplan did not see it fit to think about the issue for a significant time, and you did?
There’s also the AI researchers who have had the privilege of thinking about relevant subjects for a very long time, education, and accomplishments which verify that their thinking adds up over time—and who are largely the actual source for the opinions held by the policy makers.
By the way, note that the usual method of rejection of wrong ideas, is not even coming up with wrong ideas in the first place, and general non-engagement of wrong ideas. This is because the space of wrong ideas is much larger than the space of correct ideas.
What I expect to see in the counter-factual world where the AI risk is a big problem, is that the proponents of the AI risk in that hypothetical world have far more impressive and far more relevant accomplishments and credentials.
but to have the object-level debates about the facts at issue.
The first problem with highly speculative topics is that great many arguments exist in favour of either opinion on a speculative topic. The second problem is that each such argument relies on a huge number of implicit or explicit assumptions that are likely to be violated due to their origin as random guesses. The third problem is that there is no expectation that the available arguments would be a representative sample of the arguments in general.
This doesn’t appear to be accurate. E.g. Carl & Paul changed my mind about the probability of hard takeoff.
Hmm, I was under the impression that you weren’t a big supporter of the hard takeoff to begin with.
If I mention a public figure agreeing with me, it’s generally not because this plays a significant role in my own estimates, it’s because other people think there’s a stronger correlation between social status and correctness than I do.
Well, your confidence should be increased by the agreement; there’s nothing wrong with that. The problem is when it is not balanced by the expected decrease by disagreement.
What I expect to see in the counter-factual world where the AI risk is a big problem, is that the proponents of the AI risk in that hypothetical world have far more impressive and far more relevant accomplishments and credentials.
There are a great many differences in our world model, and I can’t talk through them all with you.
Maybe we could just make some predictions? E.g. do you expect Stephen Hawking to hook up with FHI/CSER, or not? I think… oops, we can’t use that one: he just did. (Note that this has negligible impact on my own estimates, despite him being perhaps the most famous and prestigious scientist in the world.)
Okay, well… If somebody takes a decent survey of mainstream AI people (not AGI people) about AGI timelines, do you expect the median estimate to be earlier or later than 2100? (Just kidding; I have inside information about some forthcoming surveys of this type… the median is significantly sooner than 2100.)
Okay, so… do you expect more or fewer prestigious scientists to take AI risk seriously 10 years from now? Do you expect Scott Aaronson and Peter Norvig, within 25 years, to change their minds about AI timelines, and concede that AI is fairly likely within 100 years (from now) rather than thinking that it’s probably centuries or millennia away? Or maybe you can think of other predictions to make. Though coming up with crisp predictions is time-consuming.
Well, I too expect some form of something that we would call “AI”, before 2100. I can even buy into some form of accelerating progress, albeit the progress would be accelerating before the “AI” due to the tools using relevant technologies, and would not have that sharp of a break. I even do agree that there is a certain level of risk involved in all the future progress including progress of the software.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than “Steven Hawkins hooked up with FHI” or what ever you are outlining here.
You do frequently lament that the AI risk is underfunded, under-supported, and there’s under-awareness about it. In the hypothetical world, this is not the case and you can only lament that the rational spending should be 2 billions rather than 1 billion.
edit: and of course, my true rejection is that I do not actually see rational inferences leading there. The imaginary world stuff is just a side-note to explain how non-experts generally look at it.
edit2: and I have nothing against FHI’s existence and their work. I don’t think they are very useful, or address any actual safety issues which may arise, though, but with them I am fairly certain they aren’t doing any harm either (Or at least, the possible harm would be very small). Promoting the idea that AI is possible within 100 years, however, is something that increases funding for AI all across the board.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than “Steven Hawkins hooked up with FHI” or what ever you are outlining here.
Right, this just goes back to the same disagreement in our models I was trying to address earlier by making predictions. Let me try something else, then. Here are some relevant parts of my model:
I expect most highly credentialed people to not be EAs in the first place.
I expect most highly credential people to be mostly just aware of risks they happen to have heard about (e.g. climate change, asteroids, nuclear war), rather than attempting a systematic review of risks (e.g. by reading the GCR volume).
I expect most highly credentialed people to respond fairly well when actuarial risk is easily calculated (e.g. asteroid risk), and not-so-well when it’s more difficult to calculate (e.g. many insurance companies went bankrupt after 9/11).
I expect most highly credentialed people to have spent little time on explicit calibration training.
I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
I expect most highly credentialed people to know very little about AI, and very little about AI risk.
I expect that in general, even those highly credentialed people who intuitively think AI risk is a big deal will not even contact the people who think about AI risk for a living in order to ask about their views and their reasons for them, due to basic VoI failure.
I expect most highly credentialed people to have fairly reasonable views within their own field, but to often have crazy views “outside the laboratory.”
I expect most highly credentialed people to not have a good understanding of Bayesian epistemology.
I expect most highly credentialed people to continue working on, and caring about, whatever their career has been up to that point, rather than suddenly switching career paths on the basis of new information and an EV calculation.
I expect most highly credentialed people to not understand lots of pieces of “black swan epistemology” like this one and this one.
The question should not be about “highly credentialed” people alone, but about how they fare compared to people who are rather very low “credentialed”.
In particular, on your list, I expect people with fairly low credentials to fare much worse, especially at identification of the important issues as well as on rational thinking. Those combine multiplicatively, making it exceedingly unlikely—despite the greater numbers of the credential-less masses—that people who lead the work on an important issue would have low credentials.
I expect most highly credentialed people to not be EAs in the first place.
What’s EA? Effective altruism? If it’s an existential risk, it kills everyone, selfishness suffices just fine.
e.g. many insurance companies went bankrupt after 9/11
Ohh, come on. That is in no way a demonstration that insurance companies in general follow faulty strategies, and especially is not a demonstration that you could do better.
I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
In particular, on your list, I expect people with fairly low credentials to fare much worse
No doubt! I wasn’t comparing highly credentialed people to low-credentialed people in general. I was comparing highly credentialed people to Bostrom, Yudkowsky, Shulman, etc.
But why exactly would you expect conventional researchers in AI and related technologies (also including provable software, as used in the aerospace industry, and a bunch of other topics), with credentials and/or accomplishments in said fields, to fare worse on that list’s score?
Furthermore, with regards to the rationality, risks of mistake, and such… very little was done that can be checked for correctness in a clear cut way—most is of such nature that even when wrong it would not be possible to conclusively demonstrate it wrong. The few things that can be checked… look, when you write an article like this , discussing irrationality of Enrico Fermi, there’s a substantial risk of appearing highly arrogant (and irrational) if you get the technical details wrong. It is a miniature version of AI risk problem—you need to understand the subject, and if you don’t, there’s negative consequences. It is much, much easier to not goof up in things like that, than AI direction.
As you guys are researching into actual AI technologies, the issue is that one should be able to deem your effort less of a risk. Mere “we are trying to avoid risk and we think they don’t” can’t do. The cost of a particularly bad friendly AI goof-up is a sadistic AI (to borrow the term from Omohundro). A sadistic AI can probably run far more tortured minds than a friendly AI can run minds, by a very huge factor, so the risk of a goof up must be quite a lot lower than anyone demonstrated.
BTW, I went back and numbered the items in my list so they’re easier to refer to.
But why exactly would you expect conventional researchers in AI and related technologies… with credentials and/or accomplishments in said fields, to fare worse on that list’s score?
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7)†, (8), (10), and (12), but Bostrom, Yudkowsky and Shulman rather uncontroversially do satisfy those items. I also expect B/Y/S to outperform most credentialed experts on (4), (9), and (11), but I understand that’s a subjective judgment call and it would take a long time for me to communicate my reasons.
† The AI risk part of 7, anyway. Obviously, AI people specifically know a lot about AI.
Edit: Also, I’ll briefly mention that I haven’t downvoted any of your comments in this conversation.
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7), (8), (10), and (12)
Ok, let’s go over your list, for the AI people.
1 I expect most highly credentialed people to not be EAs in the first place.
If EA is effective altruism, that’s not relevant because one doesn’t have to be an altruist to care about existential risks.
2 I expect most highly credentialed people to not be familiar with the arguments for caring about the far future.
I expect them to be able to come up with that independently if it is a good idea.
3 I expect most highly credential people to be mostly just aware of risks they happen to have heard about (e.g. climate change, asteroids, nuclear war), rather than attempting a systematic review of risks (e.g. by reading the GCR volume).
I expect intelligent people to be able to foresee risks, especially when prompted by the cultural baggage (modern variations on the theme of Golem)
4 I expect most highly credentialed people to respond fairly well when actuarial risk is easily calculated (e.g. asteroid risk), and not-so-well when it’s more difficult to calculate (e.g. many insurance companies went bankrupt after 9/11).
Well, that ought to imply some generally better ability to evaluate hard to calculate probabilities, which would imply that you guys should be able to make quite a bit of money.
5 I expect most highly credentialed people to have spent little time on explicit calibration training.
The question is how well are they calibrated, not how much time they spent. You guys see miscalibration of famous people everywhere, even in Enrico Fermi.
6 I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
Once again, how unbiased is what’s important, not how much time spent on a very specific way to acquire an ability. I expect most accomplished people to have encountered far more feedback on being right / being wrong through their education and experience.
7 I expect most highly credentialed people to know very little about AI, and very little about AI risk.
Doesn’t apply to people in AI related professions.
8 I expect that in general, even those highly credentialed people who intuitively think AI risk is a big deal will not even contact the people who think about AI risk for a living in order to ask about their views and their reasons for them, due to basic VoI failure.
The way to raise VoI is prior history of thinking about something else for a living, with impressive results.
9 I expect most highly credentialed people to have fairly reasonable views within their own field, but to often have crazy views “outside the laboratory.”
Well, less credentialed people are just like this except they don’t have a laboratory inside of which they are sane, that’s usually why they are less credentialed in the first place.
10 I expect most highly credentialed people to not have a good understanding of Bayesian epistemology.
Of your 3, I only weakly expect Bostrom to have learned the necessary fundamentals for actually applying Bayes theorem correctly in somewhat non-straightforward cases.
Yes, the basic formula is simple, but derivations are subtle and complex for non independent evidence or cases involving loops in the graph or all those other things…
It’s like arguing that you are better equipped for a job at Weta Digital than any employee there because you know quantum electrodynamics (the fundamentals of light propagation), and they’re using geometrical optics.
I expect many AI researchers to understand the relevant mathematics a lot, lot better than the 3 on your list.
And I expect credentialed people in general to have a good understanding of the variety of derivative tricks that are used to obtain effective results under uncertainty when the Bayes theorem can not be effectively applied.
11 I expect most highly credentialed people to continue working on, and caring about, whatever their career has been up to that point, rather than suddenly switching career paths on the basis of new information and an EV calculation.
Yeah, well, and I expect non-credentialed people to have too much to lose from backing out of it in the event that the studies return a negative.
12 I expect most highly credentialed people to not understand lots of pieces of “black swan epistemology” like this one and this one.
You lose me here.
I would make a different list, anyway. There’s my list:
Relevant expertise as measured by educational credentials and/or accomplishments. Expertise is required for correctly recognizing risks (e.g. an astronomer is better equipped for recognizing risks from the outer space, a physicist for recognizing faults in a nuclear power plant design, et cetera)
Proven ability to make correct inferences (largely required for 1).
Self preservation (most of us have it)
Lack of 1 is an automatic dis-qualifier in my list. It doesn’t matter how much you are into things that you think are important for identifying, say, faults in a nuclear power plant design. If you are not an engineer, a physicist, or the like, you aren’t going to qualify for that job via some list you make yourself, which conveniently omits (1).
I disagree with many of your points, but I don’t have time to reply to all that, so to avoid being logically rude I’ll at least reply to what seems to be your central point, about “relevant expertise as measured by educational credentials and/or accomplishments.”
Who has educational credentials and/or accomplishments relevant to future AGI designs or long-term tech forecasting? Also, do you particularly disagree with what I wrote in AGI Impact Experts and Friendly AI Experts?
Also, in general, I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts (e.g. facts relevant to the theses in this post). Argument screens off authority, and all that.
Edit: Also, my view of Holden Karnofsky might be illustrative. I take Holden Karnofsky more seriously than almost anyone on the cost-effectiveness of global health interventions, despite the fact that he has 0 relevant degrees, 0 papers published in relevant journals, 0 awards for global health work, etc. Degrees and papers and so on are only proxy variables for what we really care about, and are easily screened off by more relevant variables, both for the case of Karnofsky on global health and for the case of Bostrom, Yudkowsky, Shulman, etc. on AI risk.
For Karnofsky and to some extent Bostrom yes, Shulman is debatable, Yudkowsky tried to get screened (tried to write a programming language, for example, wrote a lot of articles on various topics, many of them wrong, tried to write technical papers (TDT), really badly), and failed to pass the screening by a very big margin. Entirely irrational arguments about 10% counter-factual impact of his are also a part of failure. Omohundro passed with flying colours (his PhD is almost entirely irrelevant at that point, as it is screened off by his accomplishments in AI).
I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts....
Exactly. All of this is wasted effort once either FAI or UFAI is developed.
Who has educational credentials and/or accomplishments relevant to future AGI designs or long-term tech forecasting?
There’s the more relevant accomplishments, there are less relevant accomplishments, and lacks of accomplishment.
Also, in general, I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts
I agree that a discussion of strategically relevant facts would be much more productive. I don’t see facts here. I see many speculations. I see a lot of making things up to fit the conclusion.
If I were to tell you that I can, for example, win a very high stakes programming contest (with a difficult, open problem that has many potential solutions that can be ranked in terms of quality), the discussion of my approach to the contest problem between you and me would be almost useless for your or my prediction of victory (provided that basic standards of competence are met), irrespective of whenever my idea is good. Prior track record, on the other hand, would be a good predictor. This is how it is for a very well defined problem. It is not going to be better for a less well understood problem.
If EA is effective altruism, that’s not relevant because one doesn’t have to be an altruist to care about existential risks.
‘EA’ here refers to the traits a specific community seems to exemplify (though those traits may occur outside the community). So more may be suggested than the words ‘effective’ and ‘altruism’ contain.
In terms of the terms, I think ‘altruism’ here is supposed to be an inclination to behave a certain way, not an other-privileging taste or ideology. Think ‘reciprocal altruism’. You can be an egoist who’s an EA, provided your selfish calculation has led you to the conclusion that you should devote yourself to efficiently funneling money to the world’s poorest, efficiently reducing existential risks, etc. I’m guessing by ‘EA’ Luke has in mind a set of habits of looking at existential risks that ‘Effective Altruists’ tend to exemplify, e.g., quantifying uncertainty, quantifying benefit, strongly attending to quantitative differences, trying strongly to correct for a specific set of biases (absurdity bias, status quo bias, optimism biases, availability biases), relying heavily on published evidence, scrutinizing the methodology and interpretation of published evidence....
I expect them to be able to come up with that independently if it is a good idea.
My own experience is that I independently came up with a lot of arguments from the Sequences, but didn’t take them sufficiently seriously, push them hard enough, or examine them in enough detail. There seems to be a big gap between coming up with an abstract argument for something while you’re humming in the shower, and actually living your life in a way that’s consistent with your believing the argument is sound.
My own experience is that I independently came up with a lot of arguments from the Sequences, but didn’t take them sufficiently seriously, push them hard enough, or examine them in enough detail.
But we are speaking of credentialed people. They’re fairly driven.
Furthermore, general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views. (The biggest problem with “Bayesianism” is dis-balanced/selective updates)
First off, if one can support existential risk for non Pascal’s wager type reasons then enormous utility of the future should not be relevant. If it is actually a requirement then I don’t think there’s anything to discuss here.
Secondarily, the most common norm of morality (Assuming we ignore things like Sharia), as specified in the laws of progressive countries, or as extrapolation of legal progress in less progressive ones, is to value the future people (we disapprove of smoking while pregnant), but not value counter-factual creation of future people (we allow abortion, and especially when the child would be disadvantaged and not have a fair chance). Rather than inferring the prevailing morality from the law and discussing it, various bad ideas are invented and discussed to make the argument appear stronger than it really is.
It is not that I am not exposed to this worldview. I am. It is that choosing between A: hurt someone, but a large number of happy people will be created, and B: not hurt someone, but a large number of happy people will not be created (with the deliberate choice having the causal impact on the hurting and creation), A is both illegal and immoral.
general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views.
When I hear that Joe has a new argument against a belief of mine, then my confidence in my belief lowers a bit, and my confidence in Joe’s competence also lowers a bit. If I then go on to actually evaluate the argument in detail and discover that it’s an extraordinarily poor one, this should generally increase my confidence to higher than it was before I heard that Joe had an argument, and it should further lower my confidence in Joe’s competence.
I’ve spent enough time looking at the specific arguments for and against many of these propositions to have the contents of those arguments overwhelm my expertise priors in both directions, such that I just don’t see a whole lot of value in discussing anything but the arguments themselves, when my goal (and yours) is to figure out the level of merit of the arguments.
if one can support existential risk for non Pascal’s wager type reasons then enormous utility of the future should not be relevant.
It sounds like you’re committing the Pascal’s Wager Fallacy Fallacy. If you aren’t, then I’m not understanding your point. Large future utilities should count more than small future utilities, and multiplying by low probabilities is fine if the probabilities aren’t vanishingly low.
Choosing between A: hurt someone, but a large number of happy people get created, and B: not hurt someone, but a large number of happy people do not get created, A is both illegal and immoral.
I think there’s a quantitative tradeoff between the happiness of currently existent people and the happiness of possibly-created people. A strict rule ‘Counterfactual People Have Absolutely No Value’ leads to absurd conclusions, e.g., it’s not worthwhile to create an infinite number of infinitely happy and well-off people if the cost is that your shoulder itches for a few seconds. It’s at least a little worthwhile to create people with awesome lives, even if they should get weighted less than currently existent people.
I’ve spent enough time looking at the specific arguments for and against many of these propositions to have the contents of those arguments overwhelm my expertise priors in both directions, such that I just don’t see a whole lot of value in discussing anything but the arguments themselves, when my goal (and yours) is to figure out the level of merit of the arguments.
You don’t want the outcome to be biased by the availability of the arguments, right? Really, I think you do not account for the fact that the available arguments are merely samples from the space of possible arguments (which make different speculative assumptions, in a very large space of possible speculations). Picked non uniformly, too, as arguments for one side may be more available, or their creation may maximize personal present-day utility of more agents. Individual samples can’t be particularly informative in such a situation.
It’s at least a little worthwhile to create people with awesome lives, even if they should get weighted less than currently existent people.
The issue is that the number of people you can speculate you affect grows much faster than the prior for the speculation decreases. Constant factors do not help with that, they just push the problem a little further.
A strict rule ‘Counterfactual People Have Absolutely No Value’ leads to absurd conclusions, e.g., it’s not worthwhile to create an infinite number of infinitely happy and well-off people if the cost is that your shoulder itches for a few seconds.
I don’t see that as problematic. Ponder the alternative for a moment: you may be ok with a shoulder itch, but are you OK with 10 000 years of the absolutely worst torture imaginable, for the sake of creation of 3^^^3 or 3^^^^^3 or however many really happy people? What’s about your death vs their creation?
edit: also you might have the value of those people to yourself (as potential mates and whatnot) leaking in.
It sounds like you’re committing the Pascal’s Wager Fallacy Fallacy. If you aren’t, then I’m not understanding your point. Large future utilities should count more than small future utilities, and multiplying by low probabilities is fine if the probabilities aren’t vanishingly low.
If the probabilities aren’t vanishingly low, you reach basically same conclusions without requiring extremely large utilities. 7 billion people dying is quite a lot, too. If you see extremely large utilities on a list of requirements for caring about the issue, when you already have at least 7 billion lives at stake, then it is a Pascal’s wager.
Actually, I don’t see vanishingly small probabilities problematic, I see small probabilities where the bulk of probability mass is unaccounted for, problematic. E.g. response to low risk from a specific asteroid is fine, because it’s alternative positions in space are accounted for (and you have assurance you won’t put it on an even worse trajectory)
Furthermore, general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views. (The biggest problem with “Bayesianism” is dis-balanced/selective updates)
Updating on someone else’s decision to accept or reject a position should depend on their reason for their position. Information cascades is relevant.
Yes, of course. But also keep in mind that wrong positions are often rejected by the mechanism that generates positions, rather than the mechanism that checks the generated positions.
I did see Eelco Hoogendoorn ’s and it is absolutely spot on.
I’m hardly a fan of Caplan, but he has some Bayesianism right:
Based on how things like this asymptote or fail altogether, he has a low prior for foom.
He has low expectation of being able to identify in advance (without the work equivalent to the creation of the AI) exact mechanisms by which it is going to asymptote or fail, irrespective of whenever it does or does not asymptote or fail, so not knowing such mechanisms does not bother him a whole lot.
Even assuming he is correct he expects a plenty of possible arguments against this position (which are reliant on speculations), as well as expects to see some arguers, because the space of speculative arguments is very huge. So such arguments are not going to move him anywhere.
People don’t do that explicitly any more than someone who’s playing football is doing Newtonian mechanics explicitly. Bayes theorem is no less fundamental than the laws of motion of the football.
Likewise for things like non-testability: nobody’s doing anything explicitly, it is just the case that due to something you guys call “conservation of expected evidence” , when there is no possibility of evidence against a proposition, then a possibility of evidence in favour of the proposition would violate the Bayes theorem.
when there is no possibility of evidence against a proposition, then a possibility of evidence in favour of the proposition would violate the Bayes theorem.
I’m not sure how you could have such a situation, given that absence of expected evidence is evidence of the absence. Do you have an example?
Well, the probabilities wouldn’t be literally zero. What I mean is that lack of a possibility of strong evidence against something, and only a possibility of very weak evidence against it (via absence of evidence) implies that strong evidence in favour of it must be highly unlikely. Worse, such evidence just gets lost among the more probable ‘evidence that looks strong but is not’.
Absence of evidence isn’t necessarily a weak kind of evidence.
If I tell you there’s a dragon sitting on my head, and you don’t see a dragon sitting on my head, then you can be fairly sure there’s not a dragon on my head.
On the other hand, if I tell you I’ve buried a coin somewhere in my magical 1cm deep garden—and you dig a random hole and don’t find it—not finding the coin isn’t strong evidence that I’ve not buried one. However, there there’s so much potential weak evidence against. If you’ve dug up all but a 1cm square of my garden—the coin’s either in that 1cm or I’m telling porkies, and what are the odds that—digging randomly—you wouldn’t have come across it by then? You can be fairly sure, even before digging up that square, that I’m fibbing.
Was what you meant analogous to one of those scenarios?
Yes, like the latter scenario. Note that the expected utility of digging is low when the evidence against from one dig is low.
edit: Also. In the former case, not seeing a dragon sitting on your head is very strong evidence against there being a dragon. Unless you invoke un-testable invisible dragons which may be transparent to x-rays, let dust pass through it unaffected, and so on. In which case, I should have a very low likelihood of being convinced that there is a dragon on your head, if I know that the evidence against would be very weak.
edit2: Russel’s teapot in the Kuiper belt is a better example still. When there can be only very weak evidence against it, the probability of encountering or discovering strong evidence in favour of it must be low also, making it not worth while to try to come up with evidence that there is a teapot in the Kuiper belt (due to low probability of success), even when the prior probability for the teapot is not very low.
Then, to extend the analogy: Imagine that digging has potentially negative utility as well as positive. I claim to have buried both a large number of nukes and a magical wand in the garden.
In order to motivate you to dig, you probably want some evidence of magical wands. In this context that would probably be recursively improving systems where, occasionally, local variations rapidly acquire super-dominance over their contemporaries when they reach some critical value. Evolution probably qualifies there—other bipedal frames with fingers aren’t particularly dominant over other creatures in the same way that we are, but at some point we got smart enough to make weapons (note that I’m not saying that was what intelligence was for though) and from then on, by comparison to all other macroscopic land-dwelling forms of life, we may as well have been god.
And since then that initial edge in dominance has only ever allowed us to become more dominant. Creatures afraid of wild animals are not able to create societies with guns and nuclear weapons—you’d never have the stability for long enough.
In order to motivate you not to dig, you probably want some evidence of nukes. In this context, recursively—I’m not sure improving is the right word here—systems with a feedback state, that create large amounts of negative value. Well, to a certain extent that’s a matter of perspective—from the perspective of extinct species the ascendancy of humanity would probably not be anything to cheer about, if they were in a position to appreciate it. But I suspect it can at least stand on its own that it tends to be the case that failure cascades are easier to make than cascade successes. One little thing goes wrong on your rocket and then the situation multiplies; a small error in alignment rapidly becomes a bigger one; or the timer on your patriot battery is losing a fraction of a second and over time your perception of where the missiles are is off significantly. - it’s only with significant effort that we create systems where errors don’t multiply.
(This is analogous to altering your expected value of information—like if earlier you’d said you didn’t want to dig and I’d said, ‘well there’s a million bucks there’ instead—you’d probably want some evidence that I had a million bucks, but given such evidence the information you’d gain from digging would be worth more.)
This seems to be fairly closely analogous to Elizer’s claims about AI, at least if I’ve understood them correctly, that we have to hit an extremely small target and it’s more likely that we’re going to blow ourselves to itty-bitty pieces/cover the universe in paperclips if we’re just fooling around hoping to hit on it by chance.
If you believe that such is the case, then the only people you’re going to want looking for that magic wand—if you let anyone do it at all—are specialists with particle detectors—indeed if your garden is in the middle of a city you’ll probably make it illegal for kids to play around anywhere near the potential bomb site.
Now, we may argue over quite how strongly we have to believe in the possible existence of magitech nukes to justify the cost of fencing off the garden—personally I think the statement:
if you take a thorough look at actually existing creatures, it’s not clear that smarter creatures have any tendency to increase their intelligence.
Is to constrain what you’ll accept for potential evidence pretty dramatically—we’re talking about systems in general, not just individual people, and recursively improving systems with high asymptotes relative to their contemporaries have happened before.
It’s not clear to me that the second claim he makes is even particularly meaningful:
In the real-world, self-reinforcing processes eventually asymptote. So even if smarter creatures were able to repeatedly increase their own intelligence, we should expect the incremental increases to get smaller and smaller over time, not skyrocket to infinity.
Sure, I think that they probably won’t go to infinity—but I don’t see any reason to suspect that they won’t converge on a much higher value than our own native ability. Pretty much all of our systems do, from calculators to cars.
We can even argue over how you separate the claims that something’s going to foom from the false claims of such (I’d suggest, initially, just seeing how many claims that something was going to foom have actually been made within the domain of technological artefacts, it may be that the base-line credibility is higher than we think.) But that’s a body of research that Caplan, as far as I’m aware, hasn’t forwarded. It’s not clear to me that it’s a body of research with the same order of difficulty as creating an actual AI either. And, in its absence, it’s not clear to me that to answer in effect, “I’ll believe it when I see the mushroom cloud.” is a particularly rational response.
I was mostly referring to the general lack of interest in the discussion of un-falsifiable propositions by the scientific community. The issue is that un-falsifiable proposition are also the ones for which it is unlikely that in the discussion you will be presented with evidence in favour of them.
The space of propositions is the garden I am speaking of. And digging up false propositions is not harmless.
With regards to the argument of yours, I think you vastly under-estimate the size of the high-dimensional space of possible software, and how distant in this space are the little islands of software that actually does something interesting, as distant from each other as Bolzmann minds are within our universe (Albeit, of course, depending on the basis, possible software is better clustered).
Those spatial analogies, they are a great fallacy generator, a machine for getting quantities off by mind-bogglingly huge factors. In your mental image, you have someone create those nukes and put them in the sand, for the hapless individuals to find. In the reality that’s not how you find nuke. You venture into this enormous space of possible designs, as vast as the distance from here to the closest exact replica of The Gadget which spontaneously formed from a supernova by the random movement of uranium atoms. When you have to look in the space this big, you don’t find this replica of The Gadget without knowing what you’re looking for quite well.
With regards to listing biases to help arguments, given that I have no expectation that one could not handwave up a fairly plausible bias that would work in the direction of a specific argument, the direct evidential value of listing biases in such manner, on the proposition, is zero (or an epsilon). You could have just as well argued that the individuals who are not afraid of cave bears get killed by the cave bears; there’s too much “give” in your argument for it to have any evidential value. I can freely ignore it without having to bother to come up with a balancing bias (as people like Caplan rightfully do, without really bothering to outline why).
Great quote.
Unfortunately, we find ourselves in a world where the world’s policy-makers don’t just profess that AGI safety isn’t a pressing issue, they also aren’t taking any action on AGI safety. Even generally sharp people like Bryan Caplan give disappointingly lame reasons for not caring. :(
Why won’t you update towards the possibility that they’re right and you’re wrong?
This model should rise up much sooner than some very low prior complex model where you’re a better truth finder about this topic but not any topic where truth-finding can be tested reliably*, and they’re better truth finders about topics where truth finding can be tested (which is what happens when they do their work), but not this particular topic.
(*because if you expect that, then you should end up actually trying to do at least something that can be checked because it’s the only indicator that you might possibly be right about the matters that can’t be checked in any way)
Why are the updates always in one direction only? When they disagree, the reasons are “lame” according to yourself, which makes you more sure everyone’s wrong. When they agree, they agree and that makes you more sure you are right.
It’s not so much that I’m a better truth finder, it’s that I’ve had the privilege of thinking through the issues as a core component of my full time job for the past two years, and people like Caplan only raise points that have been accounted for in my model for a long time. Also, I think the most productive way to resolve these debates is not to argue the meta-level issues about social epistemology, but to have the object-level debates about the facts at issue. So if Caplan replies to Carl’s comment and my own, then we can continue the object-level debate, otherwise… the ball’s in his court.
This doesn’t appear to be accurate. E.g. Carl & Paul changed my mind about the probability of hard takeoff. And when have I said that some public figure agreeing with me made me more sure I’m right? See also my comments here.
If I mention a public figure agreeing with me, it’s generally not because this plays a significant role in my own estimates, it’s because other people think there’s a stronger correlation between social status and correctness than I do.
Yes, but why Caplan did not see it fit to think about the issue for a significant time, and you did?
There’s also the AI researchers who have had the privilege of thinking about relevant subjects for a very long time, education, and accomplishments which verify that their thinking adds up over time—and who are largely the actual source for the opinions held by the policy makers.
By the way, note that the usual method of rejection of wrong ideas, is not even coming up with wrong ideas in the first place, and general non-engagement of wrong ideas. This is because the space of wrong ideas is much larger than the space of correct ideas.
What I expect to see in the counter-factual world where the AI risk is a big problem, is that the proponents of the AI risk in that hypothetical world have far more impressive and far more relevant accomplishments and credentials.
The first problem with highly speculative topics is that great many arguments exist in favour of either opinion on a speculative topic. The second problem is that each such argument relies on a huge number of implicit or explicit assumptions that are likely to be violated due to their origin as random guesses. The third problem is that there is no expectation that the available arguments would be a representative sample of the arguments in general.
Hmm, I was under the impression that you weren’t a big supporter of the hard takeoff to begin with.
Well, your confidence should be increased by the agreement; there’s nothing wrong with that. The problem is when it is not balanced by the expected decrease by disagreement.
There are a great many differences in our world model, and I can’t talk through them all with you.
Maybe we could just make some predictions? E.g. do you expect Stephen Hawking to hook up with FHI/CSER, or not? I think… oops, we can’t use that one: he just did. (Note that this has negligible impact on my own estimates, despite him being perhaps the most famous and prestigious scientist in the world.)
Okay, well… If somebody takes a decent survey of mainstream AI people (not AGI people) about AGI timelines, do you expect the median estimate to be earlier or later than 2100? (Just kidding; I have inside information about some forthcoming surveys of this type… the median is significantly sooner than 2100.)
Okay, so… do you expect more or fewer prestigious scientists to take AI risk seriously 10 years from now? Do you expect Scott Aaronson and Peter Norvig, within 25 years, to change their minds about AI timelines, and concede that AI is fairly likely within 100 years (from now) rather than thinking that it’s probably centuries or millennia away? Or maybe you can think of other predictions to make. Though coming up with crisp predictions is time-consuming.
Well, I too expect some form of something that we would call “AI”, before 2100. I can even buy into some form of accelerating progress, albeit the progress would be accelerating before the “AI” due to the tools using relevant technologies, and would not have that sharp of a break. I even do agree that there is a certain level of risk involved in all the future progress including progress of the software.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than “Steven Hawkins hooked up with FHI” or what ever you are outlining here.
You do frequently lament that the AI risk is underfunded, under-supported, and there’s under-awareness about it. In the hypothetical world, this is not the case and you can only lament that the rational spending should be 2 billions rather than 1 billion.
edit: and of course, my true rejection is that I do not actually see rational inferences leading there. The imaginary world stuff is just a side-note to explain how non-experts generally look at it.
edit2: and I have nothing against FHI’s existence and their work. I don’t think they are very useful, or address any actual safety issues which may arise, though, but with them I am fairly certain they aren’t doing any harm either (Or at least, the possible harm would be very small). Promoting the idea that AI is possible within 100 years, however, is something that increases funding for AI all across the board.
Right, this just goes back to the same disagreement in our models I was trying to address earlier by making predictions. Let me try something else, then. Here are some relevant parts of my model:
I expect most highly credentialed people to not be EAs in the first place.
I expect most highly credentialed people to not be familiar with the arguments for caring about the far future.
I expect most highly credential people to be mostly just aware of risks they happen to have heard about (e.g. climate change, asteroids, nuclear war), rather than attempting a systematic review of risks (e.g. by reading the GCR volume).
I expect most highly credentialed people to respond fairly well when actuarial risk is easily calculated (e.g. asteroid risk), and not-so-well when it’s more difficult to calculate (e.g. many insurance companies went bankrupt after 9/11).
I expect most highly credentialed people to have spent little time on explicit calibration training.
I expect most highly credentialed people to not systematically practice debiasing like some people practice piano.
I expect most highly credentialed people to know very little about AI, and very little about AI risk.
I expect that in general, even those highly credentialed people who intuitively think AI risk is a big deal will not even contact the people who think about AI risk for a living in order to ask about their views and their reasons for them, due to basic VoI failure.
I expect most highly credentialed people to have fairly reasonable views within their own field, but to often have crazy views “outside the laboratory.”
I expect most highly credentialed people to not have a good understanding of Bayesian epistemology.
I expect most highly credentialed people to continue working on, and caring about, whatever their career has been up to that point, rather than suddenly switching career paths on the basis of new information and an EV calculation.
I expect most highly credentialed people to not understand lots of pieces of “black swan epistemology” like this one and this one.
etc.
Luke, why are you arguing with Dmytry?
The question should not be about “highly credentialed” people alone, but about how they fare compared to people who are rather very low “credentialed”.
In particular, on your list, I expect people with fairly low credentials to fare much worse, especially at identification of the important issues as well as on rational thinking. Those combine multiplicatively, making it exceedingly unlikely—despite the greater numbers of the credential-less masses—that people who lead the work on an important issue would have low credentials.
What’s EA? Effective altruism? If it’s an existential risk, it kills everyone, selfishness suffices just fine.
Ohh, come on. That is in no way a demonstration that insurance companies in general follow faulty strategies, and especially is not a demonstration that you could do better.
Indeed.
A selfish person protecting against existential risk builds a bunker and stocks it with sixty years of foodstuffs. That doesn’t exactly help much.
For what existential risks is this actually an effective strategy?
A global pandemic that kills everyone?
The quality of life in a bunker is really damn low. Not to mention that you presumably won’t survive this particular risk in a bunker.
No doubt! I wasn’t comparing highly credentialed people to low-credentialed people in general. I was comparing highly credentialed people to Bostrom, Yudkowsky, Shulman, etc.
But why exactly would you expect conventional researchers in AI and related technologies (also including provable software, as used in the aerospace industry, and a bunch of other topics), with credentials and/or accomplishments in said fields, to fare worse on that list’s score?
Furthermore, with regards to the rationality, risks of mistake, and such… very little was done that can be checked for correctness in a clear cut way—most is of such nature that even when wrong it would not be possible to conclusively demonstrate it wrong. The few things that can be checked… look, when you write an article like this , discussing irrationality of Enrico Fermi, there’s a substantial risk of appearing highly arrogant (and irrational) if you get the technical details wrong. It is a miniature version of AI risk problem—you need to understand the subject, and if you don’t, there’s negative consequences. It is much, much easier to not goof up in things like that, than AI direction.
As you guys are researching into actual AI technologies, the issue is that one should be able to deem your effort less of a risk. Mere “we are trying to avoid risk and we think they don’t” can’t do. The cost of a particularly bad friendly AI goof-up is a sadistic AI (to borrow the term from Omohundro). A sadistic AI can probably run far more tortured minds than a friendly AI can run minds, by a very huge factor, so the risk of a goof up must be quite a lot lower than anyone demonstrated.
BTW, I went back and numbered the items in my list so they’re easier to refer to.
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7)†, (8), (10), and (12), but Bostrom, Yudkowsky and Shulman rather uncontroversially do satisfy those items. I also expect B/Y/S to outperform most credentialed experts on (4), (9), and (11), but I understand that’s a subjective judgment call and it would take a long time for me to communicate my reasons.
† The AI risk part of 7, anyway. Obviously, AI people specifically know a lot about AI.
Edit: Also, I’ll briefly mention that I haven’t downvoted any of your comments in this conversation.
Ok, let’s go over your list, for the AI people.
If EA is effective altruism, that’s not relevant because one doesn’t have to be an altruist to care about existential risks.
I expect them to be able to come up with that independently if it is a good idea.
I expect intelligent people to be able to foresee risks, especially when prompted by the cultural baggage (modern variations on the theme of Golem)
Well, that ought to imply some generally better ability to evaluate hard to calculate probabilities, which would imply that you guys should be able to make quite a bit of money.
The question is how well are they calibrated, not how much time they spent. You guys see miscalibration of famous people everywhere, even in Enrico Fermi.
Once again, how unbiased is what’s important, not how much time spent on a very specific way to acquire an ability. I expect most accomplished people to have encountered far more feedback on being right / being wrong through their education and experience.
Doesn’t apply to people in AI related professions.
The way to raise VoI is prior history of thinking about something else for a living, with impressive results.
Well, less credentialed people are just like this except they don’t have a laboratory inside of which they are sane, that’s usually why they are less credentialed in the first place.
Of your 3, I only weakly expect Bostrom to have learned the necessary fundamentals for actually applying Bayes theorem correctly in somewhat non-straightforward cases.
Yes, the basic formula is simple, but derivations are subtle and complex for non independent evidence or cases involving loops in the graph or all those other things…
It’s like arguing that you are better equipped for a job at Weta Digital than any employee there because you know quantum electrodynamics (the fundamentals of light propagation), and they’re using geometrical optics.
I expect many AI researchers to understand the relevant mathematics a lot, lot better than the 3 on your list.
And I expect credentialed people in general to have a good understanding of the variety of derivative tricks that are used to obtain effective results under uncertainty when the Bayes theorem can not be effectively applied.
Yeah, well, and I expect non-credentialed people to have too much to lose from backing out of it in the event that the studies return a negative.
You lose me here.
I would make a different list, anyway. There’s my list:
Relevant expertise as measured by educational credentials and/or accomplishments. Expertise is required for correctly recognizing risks (e.g. an astronomer is better equipped for recognizing risks from the outer space, a physicist for recognizing faults in a nuclear power plant design, et cetera)
Proven ability to make correct inferences (largely required for 1).
Self preservation (most of us have it)
Lack of 1 is an automatic dis-qualifier in my list. It doesn’t matter how much you are into things that you think are important for identifying, say, faults in a nuclear power plant design. If you are not an engineer, a physicist, or the like, you aren’t going to qualify for that job via some list you make yourself, which conveniently omits (1).
edit: list copy paste failed.
I disagree with many of your points, but I don’t have time to reply to all that, so to avoid being logically rude I’ll at least reply to what seems to be your central point, about “relevant expertise as measured by educational credentials and/or accomplishments.”
Who has educational credentials and/or accomplishments relevant to future AGI designs or long-term tech forecasting? Also, do you particularly disagree with what I wrote in AGI Impact Experts and Friendly AI Experts?
Also, in general, I’ll just remind everyone reading this that I don’t think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts (e.g. facts relevant to the theses in this post). Argument screens off authority, and all that.
Edit: Also, my view of Holden Karnofsky might be illustrative. I take Holden Karnofsky more seriously than almost anyone on the cost-effectiveness of global health interventions, despite the fact that he has 0 relevant degrees, 0 papers published in relevant journals, 0 awards for global health work, etc. Degrees and papers and so on are only proxy variables for what we really care about, and are easily screened off by more relevant variables, both for the case of Karnofsky on global health and for the case of Bostrom, Yudkowsky, Shulman, etc. on AI risk.
For Karnofsky and to some extent Bostrom yes, Shulman is debatable, Yudkowsky tried to get screened (tried to write a programming language, for example, wrote a lot of articles on various topics, many of them wrong, tried to write technical papers (TDT), really badly), and failed to pass the screening by a very big margin. Entirely irrational arguments about 10% counter-factual impact of his are also a part of failure. Omohundro passed with flying colours (his PhD is almost entirely irrelevant at that point, as it is screened off by his accomplishments in AI).
Exactly. All of this is wasted effort once either FAI or UFAI is developed.
There’s the more relevant accomplishments, there are less relevant accomplishments, and lacks of accomplishment.
I agree that a discussion of strategically relevant facts would be much more productive. I don’t see facts here. I see many speculations. I see a lot of making things up to fit the conclusion.
If I were to tell you that I can, for example, win a very high stakes programming contest (with a difficult, open problem that has many potential solutions that can be ranked in terms of quality), the discussion of my approach to the contest problem between you and me would be almost useless for your or my prediction of victory (provided that basic standards of competence are met), irrespective of whenever my idea is good. Prior track record, on the other hand, would be a good predictor. This is how it is for a very well defined problem. It is not going to be better for a less well understood problem.
‘EA’ here refers to the traits a specific community seems to exemplify (though those traits may occur outside the community). So more may be suggested than the words ‘effective’ and ‘altruism’ contain.
In terms of the terms, I think ‘altruism’ here is supposed to be an inclination to behave a certain way, not an other-privileging taste or ideology. Think ‘reciprocal altruism’. You can be an egoist who’s an EA, provided your selfish calculation has led you to the conclusion that you should devote yourself to efficiently funneling money to the world’s poorest, efficiently reducing existential risks, etc. I’m guessing by ‘EA’ Luke has in mind a set of habits of looking at existential risks that ‘Effective Altruists’ tend to exemplify, e.g., quantifying uncertainty, quantifying benefit, strongly attending to quantitative differences, trying strongly to correct for a specific set of biases (absurdity bias, status quo bias, optimism biases, availability biases), relying heavily on published evidence, scrutinizing the methodology and interpretation of published evidence....
My own experience is that I independently came up with a lot of arguments from the Sequences, but didn’t take them sufficiently seriously, push them hard enough, or examine them in enough detail. There seems to be a big gap between coming up with an abstract argument for something while you’re humming in the shower, and actually living your life in a way that’s consistent with your believing the argument is sound.
But we are speaking of credentialed people. They’re fairly driven.
Furthermore, general non acceptance of an idea is evidence that the idea is not good. You can’t seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views. (The biggest problem with “Bayesianism” is dis-balanced/selective updates)
In particular, when it comes to the interview that he linked for reasons why value the future…
First off, if one can support existential risk for non Pascal’s wager type reasons then enormous utility of the future should not be relevant. If it is actually a requirement then I don’t think there’s anything to discuss here.
Secondarily, the most common norm of morality (Assuming we ignore things like Sharia), as specified in the laws of progressive countries, or as extrapolation of legal progress in less progressive ones, is to value the future people (we disapprove of smoking while pregnant), but not value counter-factual creation of future people (we allow abortion, and especially when the child would be disadvantaged and not have a fair chance). Rather than inferring the prevailing morality from the law and discussing it, various bad ideas are invented and discussed to make the argument appear stronger than it really is.
It is not that I am not exposed to this worldview. I am. It is that choosing between A: hurt someone, but a large number of happy people will be created, and B: not hurt someone, but a large number of happy people will not be created (with the deliberate choice having the causal impact on the hurting and creation), A is both illegal and immoral.
When I hear that Joe has a new argument against a belief of mine, then my confidence in my belief lowers a bit, and my confidence in Joe’s competence also lowers a bit. If I then go on to actually evaluate the argument in detail and discover that it’s an extraordinarily poor one, this should generally increase my confidence to higher than it was before I heard that Joe had an argument, and it should further lower my confidence in Joe’s competence.
I’ve spent enough time looking at the specific arguments for and against many of these propositions to have the contents of those arguments overwhelm my expertise priors in both directions, such that I just don’t see a whole lot of value in discussing anything but the arguments themselves, when my goal (and yours) is to figure out the level of merit of the arguments.
It sounds like you’re committing the Pascal’s Wager Fallacy Fallacy. If you aren’t, then I’m not understanding your point. Large future utilities should count more than small future utilities, and multiplying by low probabilities is fine if the probabilities aren’t vanishingly low.
I think there’s a quantitative tradeoff between the happiness of currently existent people and the happiness of possibly-created people. A strict rule ‘Counterfactual People Have Absolutely No Value’ leads to absurd conclusions, e.g., it’s not worthwhile to create an infinite number of infinitely happy and well-off people if the cost is that your shoulder itches for a few seconds. It’s at least a little worthwhile to create people with awesome lives, even if they should get weighted less than currently existent people.
You don’t want the outcome to be biased by the availability of the arguments, right? Really, I think you do not account for the fact that the available arguments are merely samples from the space of possible arguments (which make different speculative assumptions, in a very large space of possible speculations). Picked non uniformly, too, as arguments for one side may be more available, or their creation may maximize personal present-day utility of more agents. Individual samples can’t be particularly informative in such a situation.
The issue is that the number of people you can speculate you affect grows much faster than the prior for the speculation decreases. Constant factors do not help with that, they just push the problem a little further.
I don’t see that as problematic. Ponder the alternative for a moment: you may be ok with a shoulder itch, but are you OK with 10 000 years of the absolutely worst torture imaginable, for the sake of creation of 3^^^3 or 3^^^^^3 or however many really happy people? What’s about your death vs their creation?
edit: also you might have the value of those people to yourself (as potential mates and whatnot) leaking in.
forgot to address this:
If the probabilities aren’t vanishingly low, you reach basically same conclusions without requiring extremely large utilities. 7 billion people dying is quite a lot, too. If you see extremely large utilities on a list of requirements for caring about the issue, when you already have at least 7 billion lives at stake, then it is a Pascal’s wager.
Actually, I don’t see vanishingly small probabilities problematic, I see small probabilities where the bulk of probability mass is unaccounted for, problematic. E.g. response to low risk from a specific asteroid is fine, because it’s alternative positions in space are accounted for (and you have assurance you won’t put it on an even worse trajectory)
Updating on someone else’s decision to accept or reject a position should depend on their reason for their position. Information cascades is relevant.
Yes, of course. But also keep in mind that wrong positions are often rejected by the mechanism that generates positions, rather than the mechanism that checks the generated positions.
After reading Robin’s exposition of Bryan’s thesis, I would disagree that his reasons are disappointingly lame.
Which could either indicate that the reasons are good or that your standards are lower than Luke’s and so trigger no disappointment.
Bryan is expressing a “standard economic intuition” but… did you see Carl’s comment reply on Caplan’s post, and also mine?
I did see Eelco Hoogendoorn ’s and it is absolutely spot on.
I’m hardly a fan of Caplan, but he has some Bayesianism right:
Based on how things like this asymptote or fail altogether, he has a low prior for foom.
He has low expectation of being able to identify in advance (without the work equivalent to the creation of the AI) exact mechanisms by which it is going to asymptote or fail, irrespective of whenever it does or does not asymptote or fail, so not knowing such mechanisms does not bother him a whole lot.
Even assuming he is correct he expects a plenty of possible arguments against this position (which are reliant on speculations), as well as expects to see some arguers, because the space of speculative arguments is very huge. So such arguments are not going to move him anywhere.
People don’t do that explicitly any more than someone who’s playing football is doing Newtonian mechanics explicitly. Bayes theorem is no less fundamental than the laws of motion of the football.
Likewise for things like non-testability: nobody’s doing anything explicitly, it is just the case that due to something you guys call “conservation of expected evidence” , when there is no possibility of evidence against a proposition, then a possibility of evidence in favour of the proposition would violate the Bayes theorem.
I’m not sure how you could have such a situation, given that absence of expected evidence is evidence of the absence. Do you have an example?
Well, the probabilities wouldn’t be literally zero. What I mean is that lack of a possibility of strong evidence against something, and only a possibility of very weak evidence against it (via absence of evidence) implies that strong evidence in favour of it must be highly unlikely. Worse, such evidence just gets lost among the more probable ‘evidence that looks strong but is not’.
Ah, I think I follow you.
Absence of evidence isn’t necessarily a weak kind of evidence.
If I tell you there’s a dragon sitting on my head, and you don’t see a dragon sitting on my head, then you can be fairly sure there’s not a dragon on my head.
On the other hand, if I tell you I’ve buried a coin somewhere in my magical 1cm deep garden—and you dig a random hole and don’t find it—not finding the coin isn’t strong evidence that I’ve not buried one. However, there there’s so much potential weak evidence against. If you’ve dug up all but a 1cm square of my garden—the coin’s either in that 1cm or I’m telling porkies, and what are the odds that—digging randomly—you wouldn’t have come across it by then? You can be fairly sure, even before digging up that square, that I’m fibbing.
Was what you meant analogous to one of those scenarios?
Yes, like the latter scenario. Note that the expected utility of digging is low when the evidence against from one dig is low.
edit: Also. In the former case, not seeing a dragon sitting on your head is very strong evidence against there being a dragon. Unless you invoke un-testable invisible dragons which may be transparent to x-rays, let dust pass through it unaffected, and so on. In which case, I should have a very low likelihood of being convinced that there is a dragon on your head, if I know that the evidence against would be very weak.
edit2: Russel’s teapot in the Kuiper belt is a better example still. When there can be only very weak evidence against it, the probability of encountering or discovering strong evidence in favour of it must be low also, making it not worth while to try to come up with evidence that there is a teapot in the Kuiper belt (due to low probability of success), even when the prior probability for the teapot is not very low.
Then, to extend the analogy: Imagine that digging has potentially negative utility as well as positive. I claim to have buried both a large number of nukes and a magical wand in the garden.
In order to motivate you to dig, you probably want some evidence of magical wands. In this context that would probably be recursively improving systems where, occasionally, local variations rapidly acquire super-dominance over their contemporaries when they reach some critical value. Evolution probably qualifies there—other bipedal frames with fingers aren’t particularly dominant over other creatures in the same way that we are, but at some point we got smart enough to make weapons (note that I’m not saying that was what intelligence was for though) and from then on, by comparison to all other macroscopic land-dwelling forms of life, we may as well have been god.
And since then that initial edge in dominance has only ever allowed us to become more dominant. Creatures afraid of wild animals are not able to create societies with guns and nuclear weapons—you’d never have the stability for long enough.
In order to motivate you not to dig, you probably want some evidence of nukes. In this context, recursively—I’m not sure improving is the right word here—systems with a feedback state, that create large amounts of negative value. Well, to a certain extent that’s a matter of perspective—from the perspective of extinct species the ascendancy of humanity would probably not be anything to cheer about, if they were in a position to appreciate it. But I suspect it can at least stand on its own that it tends to be the case that failure cascades are easier to make than cascade successes. One little thing goes wrong on your rocket and then the situation multiplies; a small error in alignment rapidly becomes a bigger one; or the timer on your patriot battery is losing a fraction of a second and over time your perception of where the missiles are is off significantly. - it’s only with significant effort that we create systems where errors don’t multiply.
(This is analogous to altering your expected value of information—like if earlier you’d said you didn’t want to dig and I’d said, ‘well there’s a million bucks there’ instead—you’d probably want some evidence that I had a million bucks, but given such evidence the information you’d gain from digging would be worth more.)
This seems to be fairly closely analogous to Elizer’s claims about AI, at least if I’ve understood them correctly, that we have to hit an extremely small target and it’s more likely that we’re going to blow ourselves to itty-bitty pieces/cover the universe in paperclips if we’re just fooling around hoping to hit on it by chance.
If you believe that such is the case, then the only people you’re going to want looking for that magic wand—if you let anyone do it at all—are specialists with particle detectors—indeed if your garden is in the middle of a city you’ll probably make it illegal for kids to play around anywhere near the potential bomb site.
Now, we may argue over quite how strongly we have to believe in the possible existence of magitech nukes to justify the cost of fencing off the garden—personally I think the statement:
Is to constrain what you’ll accept for potential evidence pretty dramatically—we’re talking about systems in general, not just individual people, and recursively improving systems with high asymptotes relative to their contemporaries have happened before.
It’s not clear to me that the second claim he makes is even particularly meaningful:
Sure, I think that they probably won’t go to infinity—but I don’t see any reason to suspect that they won’t converge on a much higher value than our own native ability. Pretty much all of our systems do, from calculators to cars.
We can even argue over how you separate the claims that something’s going to foom from the false claims of such (I’d suggest, initially, just seeing how many claims that something was going to foom have actually been made within the domain of technological artefacts, it may be that the base-line credibility is higher than we think.) But that’s a body of research that Caplan, as far as I’m aware, hasn’t forwarded. It’s not clear to me that it’s a body of research with the same order of difficulty as creating an actual AI either. And, in its absence, it’s not clear to me that to answer in effect, “I’ll believe it when I see the mushroom cloud.” is a particularly rational response.
I was mostly referring to the general lack of interest in the discussion of un-falsifiable propositions by the scientific community. The issue is that un-falsifiable proposition are also the ones for which it is unlikely that in the discussion you will be presented with evidence in favour of them.
The space of propositions is the garden I am speaking of. And digging up false propositions is not harmless.
With regards to the argument of yours, I think you vastly under-estimate the size of the high-dimensional space of possible software, and how distant in this space are the little islands of software that actually does something interesting, as distant from each other as Bolzmann minds are within our universe (Albeit, of course, depending on the basis, possible software is better clustered).
Those spatial analogies, they are a great fallacy generator, a machine for getting quantities off by mind-bogglingly huge factors. In your mental image, you have someone create those nukes and put them in the sand, for the hapless individuals to find. In the reality that’s not how you find nuke. You venture into this enormous space of possible designs, as vast as the distance from here to the closest exact replica of The Gadget which spontaneously formed from a supernova by the random movement of uranium atoms. When you have to look in the space this big, you don’t find this replica of The Gadget without knowing what you’re looking for quite well.
With regards to listing biases to help arguments, given that I have no expectation that one could not handwave up a fairly plausible bias that would work in the direction of a specific argument, the direct evidential value of listing biases in such manner, on the proposition, is zero (or an epsilon). You could have just as well argued that the individuals who are not afraid of cave bears get killed by the cave bears; there’s too much “give” in your argument for it to have any evidential value. I can freely ignore it without having to bother to come up with a balancing bias (as people like Caplan rightfully do, without really bothering to outline why).