Consider an alternative situation: “simply try to prevent your teenage daughter from having sex.” Well, actually achieving that goal takes more than just trying, and effective plans (which don’t cause massive collateral damage) are rarely simple.
It could be less important! The challenge is navigating value disagreements. Some people are willing to wait a century to make sure the future happens correctly, and others discuss how roughly 2 people die every second, which might stop once we reach the future, and others would comment that, if we delay for a century, we will be condemning them to death since we will ruin their chance of reaching the deathless future. Even among those who only care about existential risk, there are tradeoffs between different varieties of existential risk- it may be that by slowing down technological growth, we decrease our AGI risk but increase our asteroid risk.
Value disagreements are no doubt important. It depends on the discount rate. However, Bostrom has said that the biggest existential risks right now stem from human technology, so I think asteroid risk is not such a huge factor for the next century. If we expand that to the next ten thousand years then one might have to do some calculations.
If we assume a zero discount rate then the primary consideration becomes whether or not we can expect to have any impact on existential risk from AGI by putting it off. If we can lower the AGI-related existential risk by even 1% then it makes sense to delay AGI for even huge timespans assuming other risks are not increased too much. It therefore becomes very important to answer the question of whether such delays would in fact reduce AGI-related risk. Obviously it depends on the reasons for the delay. If the reason for the delay is a nuclear war that nearly annihilates humanity but we are lucky enough to slowly crawl back from the brink, I don’t see any obvious reason why AGI-related risk would be reduced at all. But if the reason for the delay includes some conscious effort to focus first on SIRCS then some risk reduction seems likely.
If you can come up with a good one, I’ll switch. I’m having trouble finding something where the risk of collateral damage is obvious (and obviously undesirable) and there are other agents with incentives to undermine the goal.
Sorry — your response indicates exactly in which way I should have been more clear.
Using “teenage daughter having sex” to stand for something “obviously undesirable” assumes a lot about your audience. For one, it assumes that your audience does not contain any sexually-active teenage women; nor any sex-positive parents of teenage women; nor any sex-positive sex-educators or queer activists; nor anyone who has had positive (and thus not “obviously undesirable”) experiences as (or with) a sexually active teenage woman. To any of the above folks, “teenage daughter having sex” communicates something not undesirable at all (assuming the sex is wanted, of course).
Going by cultural tropes, your choice of example gives the impression that your audience is made of middle-aged, middle-class, straight, socially conservative men — or at least, people who take the views of that sort of person to be normal, everyday, and unmarked. On LW, a lot of your audience doesn’t fit those assumptions: 25% of us are under 21; 17% of us are non-heterosexual; 38% of us grew up with non-theistic family values; and between 13% and 40% of us are non-monogamous, according to the 2011 survey for instance).
To be clear, I’m not concerned that you’re offending or hurting anyone with your example. Rather, if you’re trying to make a point to a general audience, you might consider drawing on examples that don’t assume so much.
As for alternatives: “Simply try to prevent your house from being robbed” perhaps? I suspect that a very small fraction of LWers are burglars or promoters of burglary.
I don’t have the goal of preventing my teenage daughter from having sex (firstly because I have no daughter yet, and secondly because the kind of people who would have such a goal often have a similar goal about younger sisters, and I don’t—indeed, I sometimes introduce single males to her); but I had no problem with pretending I had that goal for the sake of argument. Hell, even if Vaniver had said “simply try to cause more paperclips to exist” I would have pretended I had that goal.
BTW, I don’t think that is the real reason why people flinch at such examples. If Vaniver had said “try to win your next motorcycle race”—a goal that probably even fewer people share—would anyone have objected?
BTW, I don’t think that is the real reason why people flinch at such examples. If Vaniver had said “try to win your next motorcycle race”—a goal that probably even fewer people share—would anyone have objected?
I agree. I find it annoying when people pretend otherwise.
Small correction: The term “obviously undesirable” referred to the potential collateral damage from trying to prevent the daughter from having sex, not to her having sex.
As for alternatives: “Simply try to prevent your house from being robbed” perhaps? I suspect that a very small fraction of LWers are burglars or promoters of burglary.
Burglary is an integral part of my family heritage. That’s how we earned our passage to Australia. Specifically, burgaling some items a copper kettle, getting a death sentence and having it commuted to life in the prison continent.
With those kind of circumstances in mind I say burglary is ethically acceptable when, say, your family is starving but usually far too risky to be practical or advisable.
Consider an alternative situation: “simply try to prevent your teenage daughter from having sex.” Well, actually achieving that goal takes more than just trying, and effective plans (which don’t cause massive collateral damage) are rarely simple.
But even averting massive collateral damage could be less important than mitigating existential risk.
I think my above comment applies here.
It could be less important! The challenge is navigating value disagreements. Some people are willing to wait a century to make sure the future happens correctly, and others discuss how roughly 2 people die every second, which might stop once we reach the future, and others would comment that, if we delay for a century, we will be condemning them to death since we will ruin their chance of reaching the deathless future. Even among those who only care about existential risk, there are tradeoffs between different varieties of existential risk- it may be that by slowing down technological growth, we decrease our AGI risk but increase our asteroid risk.
Value disagreements are no doubt important. It depends on the discount rate. However, Bostrom has said that the biggest existential risks right now stem from human technology, so I think asteroid risk is not such a huge factor for the next century. If we expand that to the next ten thousand years then one might have to do some calculations.
If we assume a zero discount rate then the primary consideration becomes whether or not we can expect to have any impact on existential risk from AGI by putting it off. If we can lower the AGI-related existential risk by even 1% then it makes sense to delay AGI for even huge timespans assuming other risks are not increased too much. It therefore becomes very important to answer the question of whether such delays would in fact reduce AGI-related risk. Obviously it depends on the reasons for the delay. If the reason for the delay is a nuclear war that nearly annihilates humanity but we are lucky enough to slowly crawl back from the brink, I don’t see any obvious reason why AGI-related risk would be reduced at all. But if the reason for the delay includes some conscious effort to focus first on SIRCS then some risk reduction seems likely.
Would you mind switching to an example that doesn’t assume so much about your audience?
If you can come up with a good one, I’ll switch. I’m having trouble finding something where the risk of collateral damage is obvious (and obviously undesirable) and there are other agents with incentives to undermine the goal.
Sorry — your response indicates exactly in which way I should have been more clear.
Using “teenage daughter having sex” to stand for something “obviously undesirable” assumes a lot about your audience. For one, it assumes that your audience does not contain any sexually-active teenage women; nor any sex-positive parents of teenage women; nor any sex-positive sex-educators or queer activists; nor anyone who has had positive (and thus not “obviously undesirable”) experiences as (or with) a sexually active teenage woman. To any of the above folks, “teenage daughter having sex” communicates something not undesirable at all (assuming the sex is wanted, of course).
Going by cultural tropes, your choice of example gives the impression that your audience is made of middle-aged, middle-class, straight, socially conservative men — or at least, people who take the views of that sort of person to be normal, everyday, and unmarked. On LW, a lot of your audience doesn’t fit those assumptions: 25% of us are under 21; 17% of us are non-heterosexual; 38% of us grew up with non-theistic family values; and between 13% and 40% of us are non-monogamous, according to the 2011 survey for instance).
To be clear, I’m not concerned that you’re offending or hurting anyone with your example. Rather, if you’re trying to make a point to a general audience, you might consider drawing on examples that don’t assume so much.
As for alternatives: “Simply try to prevent your house from being robbed” perhaps? I suspect that a very small fraction of LWers are burglars or promoters of burglary.
I don’t have the goal of preventing my teenage daughter from having sex (firstly because I have no daughter yet, and secondly because the kind of people who would have such a goal often have a similar goal about younger sisters, and I don’t—indeed, I sometimes introduce single males to her); but I had no problem with pretending I had that goal for the sake of argument. Hell, even if Vaniver had said “simply try to cause more paperclips to exist” I would have pretended I had that goal.
BTW, I don’t think that is the real reason why people flinch at such examples. If Vaniver had said “try to win your next motorcycle race”—a goal that probably even fewer people share—would anyone have objected?
I agree. I find it annoying when people pretend otherwise.
Small correction: The term “obviously undesirable” referred to the potential collateral damage from trying to prevent the daughter from having sex, not to her having sex.
Oh. Well, that does make a little more sense.
I understand your perspective, and that’s a large part of why I like it as an example. Is AGI something that’s “obviously undesirable”?
Burglary is an integral part of my family heritage. That’s how we earned our passage to Australia. Specifically, burgaling some items a copper kettle, getting a death sentence and having it commuted to life in the prison continent.
With those kind of circumstances in mind I say burglary is ethically acceptable when, say, your family is starving but usually far too risky to be practical or advisable.