I think it is important to keep in mind that the approach currently favored here, in which your choice of meta-ethics guides your choice of decision theory, and in which your decision theory justifies your metaethics (in a kind of ouroborean epiphany of reflective equilibrium) - that approach is only one possible research direction.
There are other approaches that might be fruitful. In fact, it is far from clear to many people that the problem of preventing uFAI involves moral philosophy at all. (ETA: Or decision theory.)
To a small group, it sometimes appears that the only way of making progress is to maintain a narrow focus and to ruthlessly prune research subtrees as soon as they fall out of favor. But pruning in this way is gambling—it is an act of desperation by people who are made frantic by the ticking of the clock.
My preference (which may turn out to be a gamble too), is to ignore the ticking and to search the tree carefully with the help of a large, well-trained army of researchers.
Much depends of course on the quantity of time we have available. If the market progresses to AGI on it’s own in 10 years, our energies are probably best spent focused on a narrow set of practical alternatives.
If we have a hundred years, then perhaps we can afford to entertain several new generations of philosophers.
If the market progresses to AGI on it’s own in 10 years, our energies are probably best spent focused on a narrow set of practical alternatives.
But the problem itself seems to suggest that if you don’t solve it on its own terms, and instead try to mitigate the practical difficulties, you still lose completely. AGI is a universe-exploding A-Bomb which the mad scientists are about to test experimentally in a few decades, you can’t improve the outcome by building better shelters (or better casing for the bomb).
Yudkowsky apparently councils ignoring the ticking as well—here:
Until you can turn your back on your rivals and the ticking clock, blank them completely out of your mind, you will not be able to see what the problem itself is asking of you. In theory, you should be able to see both at the same time. In practice, you won’t.
I have argued repeatedly that the ticking is a fundamental part of the problem—and that if you ignore it, you just lose (with high probability) to those who are paying their clocks more attention. The “blank them completely out of your mind” advice seems to be an obviously-bad way of approaching the whole area.
It is unfortunate that getting more time looks very challenging. If we can’t do that, we can’t afford to dally around very much.
Yudkowsky apparently councils ignoring the ticking as well
Yes, and that comment may be the best thing he has ever written. It is a dilemma. Go too slow and the bad guys may win. Go too fast, and you may become the bad guys. For this problem, the difference between “good” and “bad” has nothing to do with good intentions.
Another analyis is that there are at least two types of possible problem:
One is the “runaway superintelligence” problem—which the SIAI seems focused on;
Another type of problem involves the preferences of only a small subset of human being respected.
The former problem has potentially more severe consequences (astronomical waste), but an engineering error like that seems pretty unlikely—at least to me.
The latter problem could still have some pretty bad consequences for many people, and seems much more probable—at least to me.
In a resource-limited world, too much attention on the first problem could easily contribute to running into the second problem.
I think it is important to keep in mind that the approach currently favored here, in which your choice of meta-ethics guides your choice of decision theory, and in which your decision theory justifies your metaethics (in a kind of ouroborean epiphany of reflective equilibrium) - that approach is only one possible research direction.
There are other approaches that might be fruitful. In fact, it is far from clear to many people that the problem of preventing uFAI involves moral philosophy at all. (ETA: Or decision theory.)
To a small group, it sometimes appears that the only way of making progress is to maintain a narrow focus and to ruthlessly prune research subtrees as soon as they fall out of favor. But pruning in this way is gambling—it is an act of desperation by people who are made frantic by the ticking of the clock.
My preference (which may turn out to be a gamble too), is to ignore the ticking and to search the tree carefully with the help of a large, well-trained army of researchers.
Much depends of course on the quantity of time we have available. If the market progresses to AGI on it’s own in 10 years, our energies are probably best spent focused on a narrow set of practical alternatives.
If we have a hundred years, then perhaps we can afford to entertain several new generations of philosophers.
But the problem itself seems to suggest that if you don’t solve it on its own terms, and instead try to mitigate the practical difficulties, you still lose completely. AGI is a universe-exploding A-Bomb which the mad scientists are about to test experimentally in a few decades, you can’t improve the outcome by building better shelters (or better casing for the bomb).
Yudkowsky apparently councils ignoring the ticking as well—here:
I have argued repeatedly that the ticking is a fundamental part of the problem—and that if you ignore it, you just lose (with high probability) to those who are paying their clocks more attention. The “blank them completely out of your mind” advice seems to be an obviously-bad way of approaching the whole area.
It is unfortunate that getting more time looks very challenging. If we can’t do that, we can’t afford to dally around very much.
Yes, and that comment may be the best thing he has ever written. It is a dilemma. Go too slow and the bad guys may win. Go too fast, and you may become the bad guys. For this problem, the difference between “good” and “bad” has nothing to do with good intentions.
Another analyis is that there are at least two types of possible problem:
One is the “runaway superintelligence” problem—which the SIAI seems focused on;
Another type of problem involves the preferences of only a small subset of human being respected.
The former problem has potentially more severe consequences (astronomical waste), but an engineering error like that seems pretty unlikely—at least to me.
The latter problem could still have some pretty bad consequences for many people, and seems much more probable—at least to me.
In a resource-limited world, too much attention on the first problem could easily contribute to running into the second problem.