A lot of AI safety seems to assume that humans are safer than they are, and that producing software that operates within a specification is harder than it is. It’s nice to see this paper moving towards integrating actual safety analysis (the remark about collapsing bridges was a breath of fresh air), instead of general demands that ‘the AI always do as humans say’!
A human intelligence placed in charge of a nation state can kill 7 logs of humans and still be remembered heroically. An AI system placed in charge of a utopian reshaping of the society of a major country with a ‘keep the deaths within 6 logs’ guideline that it can actually stay within would be an improvement on the status quo.
If safety people are saying ‘we cant build AI systems that could make people feel bad, and we definitely can’t build systems that kill people’ their demand for perfection is in conflict with improvement.*
I suspect that major AI alignment failure will come from ‘we put the human in charge, and human error led to the model doing bad’. The industrial/aviation safety community now rightly views ‘pilot error’ as a lazy way of ending an analysis and avoiding making the engineering changes to the system that the accident conditions demand.
*edit: imagine if the ‘airplane safety’ community had developed in 1905 (soon humans will be flying in planes!) and had resembled “AI safety” Not one human can be risked! No making planes that can carry bombs! The people who said pregnant women shouldn’t ride trains because the baby will fly out of their bodies were wrong there, but keep them off the planes!
A lot of AI safety seems to assume that humans are safer than they are, and that producing software that operates within a specification is harder than it is. It’s nice to see this paper moving towards integrating actual safety analysis (the remark about collapsing bridges was a breath of fresh air), instead of general demands that ‘the AI always do as humans say’!
A human intelligence placed in charge of a nation state can kill 7 logs of humans and still be remembered heroically. An AI system placed in charge of a utopian reshaping of the society of a major country with a ‘keep the deaths within 6 logs’ guideline that it can actually stay within would be an improvement on the status quo.
If safety people are saying ‘we cant build AI systems that could make people feel bad, and we definitely can’t build systems that kill people’ their demand for perfection is in conflict with improvement.*
I suspect that major AI alignment failure will come from ‘we put the human in charge, and human error led to the model doing bad’. The industrial/aviation safety community now rightly views ‘pilot error’ as a lazy way of ending an analysis and avoiding making the engineering changes to the system that the accident conditions demand.
*edit: imagine if the ‘airplane safety’ community had developed in 1905 (soon humans will be flying in planes!) and had resembled “AI safety” Not one human can be risked! No making planes that can carry bombs! The people who said pregnant women shouldn’t ride trains because the baby will fly out of their bodies were wrong there, but keep them off the planes!