At least with current technologies, I expect serious risks to start occuring during training, not deployment. That’s ultimately when you will the greatest learning happening, when you have the greatest access to compute, and when you will first cross the threshold of intelligence that will make the system actually dangerous. So I don’t think that just checking things after they are trained is safe.
I’m under the impression that an AGI would be monitored *during* training as well. So you’d effectively need the system to turn “evil” (utility function flipped) during the training process, and the system to be smart enough to conceal that the error occurred. So it’d need to happen a fair bit into the training process. I guess that’s possible, but IDK how likely it’d be.
Yeah, I do think it’s likely that AGI would be monitored during training, but the specific instance of Open AI staff being asleep while we train the AI is a clear instance of us not monitoring the AI during the most crucial periods (which, to be clear, I think is fine since I think the risks were indeed quite low, and I don’t see this as providing super much evidence about Open AI’s future practices)
At least with current technologies, I expect serious risks to start occuring during training, not deployment. That’s ultimately when you will the greatest learning happening, when you have the greatest access to compute, and when you will first cross the threshold of intelligence that will make the system actually dangerous. So I don’t think that just checking things after they are trained is safe.
I’m under the impression that an AGI would be monitored *during* training as well. So you’d effectively need the system to turn “evil” (utility function flipped) during the training process, and the system to be smart enough to conceal that the error occurred. So it’d need to happen a fair bit into the training process. I guess that’s possible, but IDK how likely it’d be.
Yeah, I do think it’s likely that AGI would be monitored during training, but the specific instance of Open AI staff being asleep while we train the AI is a clear instance of us not monitoring the AI during the most crucial periods (which, to be clear, I think is fine since I think the risks were indeed quite low, and I don’t see this as providing super much evidence about Open AI’s future practices)