My key point: I strongly agree with (my perception of) the intuitions behind this post, but I wouldn’t agree with the framing. Or with the title. So I think it would be really good to come up with a framing of these intuitions that wouldn’t be controversial.
To illustrate on an example: Suppose I want to use a metal rod for a particular thingy in a car. And there is some safety standard for this, and the metal rod meets this standard.[1] And now suppose you instead have that same metal rod, except the safety standard does not exist. I expect most people to argue that your your car will be exactly as safe in both cases.
Now, the fact that the rod is equally safe in the two cases does not mean that using it in the two cases is equally smart. Indeed, my personal view is that without the safety standard, using the rod for your car would be quite dumb. (And using the metaphorical rod for your superintelligent AI would be f.....g insane.) But I think that saying things like “any model without safety standards is unsafe” is misleading, and likely to be unproductive when communicating with people who have a different mindset.[2]
Then you might perhaps say that “the metal rod is safe for the purpose of being used for a particular thingy in a car”? However, this itself does not guarantee that the metal rod is actually safe for this purpose—for that to be true, you additionally need the assumption that the safety standard is “reasonable”. Where “reasonable” is undefined. And if you try to define it, you will eventually either need to rely on an informal/undefined/common-sense definition somewhere, or you need to figure out how to formalise literally everything. I am not giving up on that latter goal, but we aren’t there yet :-).
Personally, I think that being in any sort of decision-making or high-impact role while having such “different mindset” is analogous to driving a bus full of people without having a driver’s license. However, what I think about this makes no meaningful impact on how things work in the real world...
I think it would be really good to come up with a framing of these intuitions that wouldn’t be controversial.
That seems great, I’d be very happy for someone to write this up more clearly. My key point was about people’s claims and confidence about safety, and yes, clearly that was communicated less well than I hoped.
My key point: I strongly agree with (my perception of) the intuitions behind this post, but I wouldn’t agree with the framing. Or with the title. So I think it would be really good to come up with a framing of these intuitions that wouldn’t be controversial.
To illustrate on an example: Suppose I want to use a metal rod for a particular thingy in a car. And there is some safety standard for this, and the metal rod meets this standard.[1] And now suppose you instead have that same metal rod, except the safety standard does not exist. I expect most people to argue that your your car will be exactly as safe in both cases.
Now, the fact that the rod is equally safe in the two cases does not mean that using it in the two cases is equally smart. Indeed, my personal view is that without the safety standard, using the rod for your car would be quite dumb. (And using the metaphorical rod for your superintelligent AI would be f.....g insane.) But I think that saying things like “any model without safety standards is unsafe” is misleading, and likely to be unproductive when communicating with people who have a different mindset.[2]
Then you might perhaps say that “the metal rod is safe for the purpose of being used for a particular thingy in a car”? However, this itself does not guarantee that the metal rod is actually safe for this purpose—for that to be true, you additionally need the assumption that the safety standard is “reasonable”. Where “reasonable” is undefined. And if you try to define it, you will eventually either need to rely on an informal/undefined/common-sense definition somewhere, or you need to figure out how to formalise literally everything. I am not giving up on that latter goal, but we aren’t there yet :-).
Personally, I think that being in any sort of decision-making or high-impact role while having such “different mindset” is analogous to driving a bus full of people without having a driver’s license. However, what I think about this makes no meaningful impact on how things work in the real world...
That seems great, I’d be very happy for someone to write this up more clearly. My key point was about people’s claims and confidence about safety, and yes, clearly that was communicated less well than I hoped.