Shane, I used “maximizing chances of success” interchangeably as a result of treating the project as a binary pass/fail setup, for the reasons mentioned in my second reply: safety is a very small target, if you are a little bit off the mark, you miss it completely. If “working on safety” means developing FAI based on an AGI design (halting the deployment of that AGI), there is nothing wrong with that (and it’d be the only way to survive, another question is how useful that AGI design would be for FAI). Basically, I defended the position that it’s vanishingly unlikely to produce FAI without good understanding of why this particular (modified) AGI is FAI, and this understanding won’t appear at last minute, even if you have a working AGI design. Trying to tinker with that AGI won’t improve your chances if you don’t go all the way, in which case phrase “maximizing safety” won’t reflect what you did. You can’t improve safety of that AGI without fully solving the problem of FAI. Chances of winning this race in the first place, from the current situation of uncertainty, are better.
P.S. I believe metaphors I used have a more or less clear technical meaning. For example, Nature not caring about your plan means that plan won’t succeed, and the extent to which it’s morally wrong for it to fail doesn’t figure into probability of success. These are rhetoric devices to avoid known failure modes in intuitive judgment, not necessarily statements about specific errors, their presence or origin.
Shane, I used “maximizing chances of success” interchangeably as a result of treating the project as a binary pass/fail setup, for the reasons mentioned in my second reply: safety is a very small target, if you are a little bit off the mark, you miss it completely. If “working on safety” means developing FAI based on an AGI design (halting the deployment of that AGI), there is nothing wrong with that (and it’d be the only way to survive, another question is how useful that AGI design would be for FAI). Basically, I defended the position that it’s vanishingly unlikely to produce FAI without good understanding of why this particular (modified) AGI is FAI, and this understanding won’t appear at last minute, even if you have a working AGI design. Trying to tinker with that AGI won’t improve your chances if you don’t go all the way, in which case phrase “maximizing safety” won’t reflect what you did. You can’t improve safety of that AGI without fully solving the problem of FAI. Chances of winning this race in the first place, from the current situation of uncertainty, are better.
P.S. I believe metaphors I used have a more or less clear technical meaning. For example, Nature not caring about your plan means that plan won’t succeed, and the extent to which it’s morally wrong for it to fail doesn’t figure into probability of success. These are rhetoric devices to avoid known failure modes in intuitive judgment, not necessarily statements about specific errors, their presence or origin.