It doesn’t matter how small a chance of winning there is, as long as our actions improve it.
Yes, if we’re talking about the overall chance of winning, but I was talking about the chance of winning through a specific scenario (directly building FAI). If the chance of that is tiny, why did your cost/benefit analysis of the proposed course of action (encouraging open FAI research) focus completely on it? Shouldn’t we be thinking more about how the proposal affects other ways of winning? ETA: To spell it out, encouraging open FAI research decreases the probability that we win by winning the WBE race or through intelligence amplification, by increasing the probability that UFAI happens first.
Giving up doesn’t seem like a strategy that leads to winning.
Nobody is saying “let’s give up”. If we don’t encourage open FAI research, we can still push for a positive Singularity in other ways, some of which I’ve posted about recently in discussion.
The strategy of navigating the WBE transition (or some more speculative intelligence improvement tool) is a more complicated question, and I don’t see in what way the background catastrophic risk matters for it.
What do you mean? What aren’t you seeing?
The same action can make immediate risk worse, but probability of eventually winning higher.
Yes, of course. I am talking about the probability of eventually winning.
Yes, if we’re talking about the overall chance of winning, but I was talking about the chance of winning through a specific scenario (directly building FAI). If the chance of that is tiny, why did your cost/benefit analysis of the proposed course of action (encouraging open FAI research) focus completely on it?
I see, I’m guessing you view the “second round” (post-WBE/human intelligence improvement) as not being similarly unlikely to eventually win. I agree that if the first round (working on FAI now, pre-WBE) has only a tiny chance of winning, while the second has a non-tiny chance (taking into account the probability of no catastrophe till the second round and it being dominated by a FAI project rather than random AGI), then it’s better to sacrifice the first round to make the second round healthier. But I also only see a tiny chance of winning the second round, mostly because of the increasing UFAI risk and the difficulty of winning a race that grants you the advantages of the second round, rather than producing an UFAI really fast.
Yes, if we’re talking about the overall chance of winning, but I was talking about the chance of winning through a specific scenario (directly building FAI). If the chance of that is tiny, why did your cost/benefit analysis of the proposed course of action (encouraging open FAI research) focus completely on it? Shouldn’t we be thinking more about how the proposal affects other ways of winning? ETA: To spell it out, encouraging open FAI research decreases the probability that we win by winning the WBE race or through intelligence amplification, by increasing the probability that UFAI happens first.
Nobody is saying “let’s give up”. If we don’t encourage open FAI research, we can still push for a positive Singularity in other ways, some of which I’ve posted about recently in discussion.
What do you mean? What aren’t you seeing?
Yes, of course. I am talking about the probability of eventually winning.
(Another thread of this conversation is here.)
I see, I’m guessing you view the “second round” (post-WBE/human intelligence improvement) as not being similarly unlikely to eventually win. I agree that if the first round (working on FAI now, pre-WBE) has only a tiny chance of winning, while the second has a non-tiny chance (taking into account the probability of no catastrophe till the second round and it being dominated by a FAI project rather than random AGI), then it’s better to sacrifice the first round to make the second round healthier. But I also only see a tiny chance of winning the second round, mostly because of the increasing UFAI risk and the difficulty of winning a race that grants you the advantages of the second round, rather than producing an UFAI really fast.