What happens “by default” typically has very much to do with what actually happens the first few times, unless extraordinary levels of caution and preparation are applied.
Things like commercial flight and the space program show that reasonable levels of caution can be routinely applied when lives are at stake.
The usual situation with engineering is that you can have whatever level of safety you are prepared to pay for.
As I understand it, most lives are currently lost to engineering through society-approved tradeoffs—in the form of motor vehicle accidents. We know how to decrease the death rate there, but getting from A to B rapidly is widely judged to be more important.
It is easy to imagine how machine intelligence is likely to produce similar effects—via unemployment. We could rectify such effects via a welfare state, but it isn’t clear how popular that willl be. We can—pretty easily—see this one coming. If the concern is with human lives, we can see now that we will need to make sure that unemployed humans have a robust safety net—and that’s a relatively straightforwards political issue.
If you have a system in which a single failure could mean extinction of anything very important, then it seems likey that there must have been many failures in safety systems and backups leading up to that situation—which would seem to count against the idea of a single failure. We have had many millions of IT failures so far already.
What happens “by default” typically has very much to do with what actually happens the first few times, unless extraordinary levels of caution and preparation are applied.
Things like commercial flight and the space program show that reasonable levels of caution can be routinely applied when lives are at stake.
The usual situation with engineering is that you can have whatever level of safety you are prepared to pay for.
As I understand it, most lives are currently lost to engineering through society-approved tradeoffs—in the form of motor vehicle accidents. We know how to decrease the death rate there, but getting from A to B rapidly is widely judged to be more important.
It is easy to imagine how machine intelligence is likely to produce similar effects—via unemployment. We could rectify such effects via a welfare state, but it isn’t clear how popular that willl be. We can—pretty easily—see this one coming. If the concern is with human lives, we can see now that we will need to make sure that unemployed humans have a robust safety net—and that’s a relatively straightforwards political issue.
If you accept that a single failure could mean extinction and worse the history of rockets and powered flight isn’t exactly inspiring.
If you have a system in which a single failure could mean extinction of anything very important, then it seems likey that there must have been many failures in safety systems and backups leading up to that situation—which would seem to count against the idea of a single failure. We have had many millions of IT failures so far already.