rhollerith raised some reasonable objections to this response that I’d like to see answered, but I’ll try and answer your question without that information:
What would you consider to be the minimum estimate of the probability that I’m right, necessary to “reasonably” motivate concern or action?
As as far as concern goes, I think my threshold for concern over your proposition is identical to my threshold for concern over UFAI, as they postulate similar results (UFAI still seems marginally worse due to the chance of destroying intelligent alien life, but I’ll write this off as entirely negligible for the current discussion). I’d say 1:10,000 is a reasonable threshold for concern of the vocalized form, “hey, is anyone looking into this?” I’d love to see some more concrete discussion on this.
“Action” in your scenario is complicated by its direct opposition to acceptance of UFAI, so I can only give you some rough constraints. To simplify, I’ll assume all risks allow equally effective action to compensate for them, even though this is clearly not the case.
Let R = the scenario you’ve described, E = the scenario in which UFAI is a credible threat. “R and E” could be described as “damned if we do, damned if we don’t”, in which case action is basically futile, so I’ll consider the case where R and E are disjoint. In that case, action would only be justifiable if p(R) > p(E). My intuition says that such justification is proportional to p(R) - p(E), but I’d prefer more clarity in this step.
So that’s a rough answer… if T is my threshold probability for action in the face of existential risk, T (p(R) - p(E)) is my threshold for action on your scenario. If R and E aren’t disjoint, it looks something like T (p(R and ~E) - p(E and ~R)).
Though I’m not convinced “R and E” necessarily means “damned either way”. If I believed E in addition to R, I think what I would do is:
Forget about memetics in either direction as likely to do more harm than good, and concentrate all available resources on developing Friendly AI as reliably and quickly as possible.
However, provably Friendly AI is still not possible with 2009 vintage tools.
So I’d do it in stages, a series of self improving AIs, the early ones with low intelligence and crude Friendliness architecture, using them to develop better Friendliness architecture in tandem with increasing intelligence for the later ones. No guarantees, but if recursive self-improvement actually worked, I think that approach would have a reasonable chance of success.
rhollerith raised some reasonable objections to this response that I’d like to see answered, but I’ll try and answer your question without that information:
As as far as concern goes, I think my threshold for concern over your proposition is identical to my threshold for concern over UFAI, as they postulate similar results (UFAI still seems marginally worse due to the chance of destroying intelligent alien life, but I’ll write this off as entirely negligible for the current discussion). I’d say 1:10,000 is a reasonable threshold for concern of the vocalized form, “hey, is anyone looking into this?” I’d love to see some more concrete discussion on this.
“Action” in your scenario is complicated by its direct opposition to acceptance of UFAI, so I can only give you some rough constraints. To simplify, I’ll assume all risks allow equally effective action to compensate for them, even though this is clearly not the case.
Let R = the scenario you’ve described, E = the scenario in which UFAI is a credible threat. “R and E” could be described as “damned if we do, damned if we don’t”, in which case action is basically futile, so I’ll consider the case where R and E are disjoint. In that case, action would only be justifiable if p(R) > p(E). My intuition says that such justification is proportional to p(R) - p(E), but I’d prefer more clarity in this step.
So that’s a rough answer… if T is my threshold probability for action in the face of existential risk, T (p(R) - p(E)) is my threshold for action on your scenario. If R and E aren’t disjoint, it looks something like T (p(R and ~E) - p(E and ~R)).
A fair answer, thanks.
Though I’m not convinced “R and E” necessarily means “damned either way”. If I believed E in addition to R, I think what I would do is:
Forget about memetics in either direction as likely to do more harm than good, and concentrate all available resources on developing Friendly AI as reliably and quickly as possible.
However, provably Friendly AI is still not possible with 2009 vintage tools.
So I’d do it in stages, a series of self improving AIs, the early ones with low intelligence and crude Friendliness architecture, using them to develop better Friendliness architecture in tandem with increasing intelligence for the later ones. No guarantees, but if recursive self-improvement actually worked, I think that approach would have a reasonable chance of success.