Let’s try a simple calculation. What is the expected FAI/UFAI ratio when friendliness is not proven? According to Eliezer’s reply in this thread, it’s close to zero:
your conditionally independent failure probabilities add up to 1 and you’re 100% doomed.
So let’s overestimate it as 1 in a million, as opposed to a more EY-like estimate of 1 in a gazillion. Of course, the more realistic odds would be dominated by an estimate that this estimate is wrong (e.g. that Eliezer is overly pessimistic), but I’m yet to see him to account for that, so let’s keep the 1-in-a-million estimate.
What are the odds of an AGI development group that is designing a self-improving AGI without provability to finish before any FAI group? It’s probably way over 99%, given that provability appears to be the hard part. But let’s be generous and make 1% (say, because the SI group thinks that they are so much ahead of everyone else in the field).
What are the odds of success of SI actively working to slow down F-less AGI development long enough to develop a provably FAI (by, say, luring away the most promising best talent in the field, UFAI x-risk awareness education, or by other means)? Unless they are way less than (1 in a million)/1% = 0.01%, it makes sense to allocate a sizable chunk of the budget to thwarting non-provably FAI efforts, with the most effective strategies prioritized. What strategies are estimated to be effective, I have no idea. Assuming education and hiring has been estimated to be better than subversion or anything else dark-artsy, we should see the above-board efforts taking a good chunk of the budget. If these efforts are only marginal, then either SI sucks at Bayesianism or it is channeling the prevention resources elsewhere (like privately convincing people to not work on “dangerous projects”). Or the above calculation is way off.
Even evil creator of AI needs somekind of controll over his child, that could be called friendliness to one person. So any group which is seriosly creating AGI and going to use it it in any efforts should be interested in FAI theory. So it could be enough to explain to any one who create AGI that he needs somekind of F-theory and it should be mathematically proven.
Most people whose paycheck comes from designing a bomb have no trouble rationalizing it. Similarly, if your paycheck depends on the AGI progress and not FAI progress, you will likely be unwilling to slow down or halt the AGI development progress, and if you are, you get fired and replaced.
I wanted to say that anyone who is creating AGI need to control it some how and by this need somekind of analog of FAI, at least for not to be killed himself. And this idea could be promoted to any AGI reasearch group.
Let’s try a simple calculation. What is the expected FAI/UFAI ratio when friendliness is not proven? According to Eliezer’s reply in this thread, it’s close to zero:
your conditionally independent failure probabilities add up to 1 and you’re 100% doomed.
So let’s overestimate it as 1 in a million, as opposed to a more EY-like estimate of 1 in a gazillion
Ignoring the issue of massive overconfidence, why do you even think these concepts are clearly enough defined to assign probability estimates to them like this? It seems pretty clear that they are not. Before discussing the probability of a poorly-defined class of events, it is best to try and say what it is that you are talking about.
Well obviously you can assign probabilities to anything—but if the event is sufficiently vague, doing so in public is rather pointless—since no one else will know what event you are talking about.
I see that others have made the same complaint in this thread—e.g. Richard Loosemore:
before deciding exactly how many angels can dance on the head of a pin, you have to make sure the “angel” concept is meaningful enough that questions about angels are meaningful
Let’s try a simple calculation. What is the expected FAI/UFAI ratio when friendliness is not proven? According to Eliezer’s reply in this thread, it’s close to zero:
So let’s overestimate it as 1 in a million, as opposed to a more EY-like estimate of 1 in a gazillion. Of course, the more realistic odds would be dominated by an estimate that this estimate is wrong (e.g. that Eliezer is overly pessimistic), but I’m yet to see him to account for that, so let’s keep the 1-in-a-million estimate.
What are the odds of an AGI development group that is designing a self-improving AGI without provability to finish before any FAI group? It’s probably way over 99%, given that provability appears to be the hard part. But let’s be generous and make 1% (say, because the SI group thinks that they are so much ahead of everyone else in the field).
What are the odds of success of SI actively working to slow down F-less AGI development long enough to develop a provably FAI (by, say, luring away the most promising best talent in the field, UFAI x-risk awareness education, or by other means)? Unless they are way less than (1 in a million)/1% = 0.01%, it makes sense to allocate a sizable chunk of the budget to thwarting non-provably FAI efforts, with the most effective strategies prioritized. What strategies are estimated to be effective, I have no idea. Assuming education and hiring has been estimated to be better than subversion or anything else dark-artsy, we should see the above-board efforts taking a good chunk of the budget. If these efforts are only marginal, then either SI sucks at Bayesianism or it is channeling the prevention resources elsewhere (like privately convincing people to not work on “dangerous projects”). Or the above calculation is way off.
Even evil creator of AI needs somekind of controll over his child, that could be called friendliness to one person. So any group which is seriosly creating AGI and going to use it it in any efforts should be interested in FAI theory. So it could be enough to explain to any one who create AGI that he needs somekind of F-theory and it should be mathematically proven.
Most people whose paycheck comes from designing a bomb have no trouble rationalizing it. Similarly, if your paycheck depends on the AGI progress and not FAI progress, you will likely be unwilling to slow down or halt the AGI development progress, and if you are, you get fired and replaced.
I wanted to say that anyone who is creating AGI need to control it some how and by this need somekind of analog of FAI, at least for not to be killed himself. And this idea could be promoted to any AGI reasearch group.
Ignoring the issue of massive overconfidence, why do you even think these concepts are clearly enough defined to assign probability estimates to them like this? It seems pretty clear that they are not. Before discussing the probability of a poorly-defined class of events, it is best to try and say what it is that you are talking about.
Feel free to explain why it is not OK to assign probabilities in this case. Clearly EY does not shy away from doing so, as the quote indicates.
Well obviously you can assign probabilities to anything—but if the event is sufficiently vague, doing so in public is rather pointless—since no one else will know what event you are talking about.
I see that others have made the same complaint in this thread—e.g. Richard Loosemore: