We’ve evaluated GPT-4o according to our Preparedness Framework and in line with our voluntary commitments. Our evaluations of cybersecurity, CBRN, persuasion, and model autonomy show that GPT-4o does not score above Medium risk in any of these categories. This assessment involved running a suite of automated and human evaluations throughout the model training process. We tested both pre-safety-mitigation and post-safety-mitigation versions of the model, using custom fine-tuning and prompts, to better elicit model capabilities.
GPT-4o has also undergone extensive external red teaming with 70+ external experts in domains such as social psychology, bias and fairness, and misinformation to identify risks that are introduced or amplified by the newly added modalities. We used these learnings to build out our safety interventions in order to improve the safety of interacting with GPT-4o. We will continue to mitigate new risks as they’re discovered.
[Edit after Simeon replied: I disagree with your interpretation that they’re being intentionally very deceptive. But I am annoyed by (1) them saying “We’ve evaluated GPT-4o according to our Preparedness Framework” when the PF doesn’t contain specific evals and (2) them taking credit for implementing their PF when they’re not meeting its commitments.]
Right. Thanks for putting the full context. Voluntary commitments refers to the WH commitments which are much narrower than the PF so I think my observation holds.
Full quote:
[Edit after Simeon replied: I disagree with your interpretation that they’re being intentionally very deceptive. But I am annoyed by (1) them saying “We’ve evaluated GPT-4o according to our Preparedness Framework” when the PF doesn’t contain specific evals and (2) them taking credit for implementing their PF when they’re not meeting its commitments.]
Right. Thanks for putting the full context. Voluntary commitments refers to the WH commitments which are much narrower than the PF so I think my observation holds.