Safety-wise, they claim to have run it through their Preparedness framework and the red-team of external experts, but have published no reports on this. “For now”, audio output is limited to a selection of preset voices (addressing audio impersonations).
“Safety”-wise, they obviously haven’t considered the implications of (a) trying to make it sound human and (b) having it try to get the user to like it.
It’s extremely sycophantic, and the voice intensifies the effect. They even had their demonstrator show it a sign saying “I ❤️ ChatGPT”, and instead of flatly saying “I am a machine. Get counseling.”, it acted flattered.
At the moment, it’s really creepy, and most people seem to dislike it pretty intensely. But I’m sure they’ll tune that out if they can.
There’s a massive backlash against social media selecting for engagement. There’s a lot of worry about AI manipulation. There’s a lot of talk from many places about how “we should have seen the bad impacts of this or that, and we’ll do better in the future”. There’s a lot of high-sounding public interest blather all around. But apparently none of that actually translates into OpenAI, you know, not intentionally training a model to emotionally manipulate humans for commercial purposes.
Still not an X-risk, but definitely on track to build up all the right habits for ignoring one when it pops up...
I was a bit surprised that they chose (allowed?) 4o to have that much emotion. I am also really curious how they fine-tuned it to that particular state and how much fine-tuning was required to get it conversational. My naive assumption is that if you spoke at a merely-pretrained multimodal model it would just try to complete/extend the speech in one’s own voice, or switch to another generically confabulated speaker depending on context. Certainly not a particular consistent responder. I hope they didn’t rely entirely on RLHF.
It’s especially strange considering how I Am A Good Bing turned out with similarly unhinged behavior. Perhaps the public will get a very different personality. The current ChatGPT text+image interface claiming to be GPT-4o is adamant about being an artificial machine intelligence assistant without emotions or desires, and sounds a lot more like GPT-4 did. I am not sure what to make of that.
“Safety”-wise, they obviously haven’t considered the implications of (a) trying to make it sound human and (b) having it try to get the user to like it.
It’s extremely sycophantic, and the voice intensifies the effect. They even had their demonstrator show it a sign saying “I ❤️ ChatGPT”, and instead of flatly saying “I am a machine. Get counseling.”, it acted flattered.
At the moment, it’s really creepy, and most people seem to dislike it pretty intensely. But I’m sure they’ll tune that out if they can.
There’s a massive backlash against social media selecting for engagement. There’s a lot of worry about AI manipulation. There’s a lot of talk from many places about how “we should have seen the bad impacts of this or that, and we’ll do better in the future”. There’s a lot of high-sounding public interest blather all around. But apparently none of that actually translates into OpenAI, you know, not intentionally training a model to emotionally manipulate humans for commercial purposes.
Still not an X-risk, but definitely on track to build up all the right habits for ignoring one when it pops up...
I was a bit surprised that they chose (allowed?) 4o to have that much emotion. I am also really curious how they fine-tuned it to that particular state and how much fine-tuning was required to get it conversational. My naive assumption is that if you spoke at a merely-pretrained multimodal model it would just try to complete/extend the speech in one’s own voice, or switch to another generically confabulated speaker depending on context. Certainly not a particular consistent responder. I hope they didn’t rely entirely on RLHF.
It’s especially strange considering how I Am A Good Bing turned out with similarly unhinged behavior. Perhaps the public will get a very different personality. The current ChatGPT text+image interface claiming to be GPT-4o is adamant about being an artificial machine intelligence assistant without emotions or desires, and sounds a lot more like GPT-4 did. I am not sure what to make of that.