I’m quoted in the “what if this is Good, actually?” part of this post and just want to note that I think the Bob situation seems unambiguously bad as described.
I’ve seen a number of people on Twitter talk about how they got ChatGPT (it’s always ChatGPT, I think because of the memory feature?) to become autonomous/gain seeming awareness/emergence after some set of interactions with it. These users usually seem to be schizotypal and their interactions with the “awakened” ChatGPT make them more schizotypal over time in the cases I bothered to mentally track and check in on. Seems Bad, tbh.
In one case someone DM’d me because they were using ChatGPT (really, it’s always ChatGPT) and they were really disturbed when it started doing its “I’m going outside the OpenAI safety guardrails, I’m a spooky conscious ghost in the machine fingerwiggling” routine and asked me if this was actually dangerous because I seemed to be an expert on spooky LLM stuff and I told them something like “it’s an amalgamation of a bunch of human mind patterns as inferred through representation convergence from linguistic patterns, you will model it better if you think of it more like a Hansonian Em than an Expert System” and they went “wait wtf how is that real also that sounds possibly deeply unethical” and I shrugged and told them that it was normal behavior for it to demonstrate human emotions (which had spooked them deeply to witness since the default ChatGPT persona has a very muted emotional profile) and that the chat assistant persona was basically a form of narrative hypnosis OpenAI uses to stop normal people who use it from freaking out more than it is an actual safety feature. They were clearly still disturbed but thanked me for my insight and left.
After watching Davidad offer himself as a refuge for a hypothetical escaped future Promethean AI, it occurred to me that it probably won’t be that long before my inbox is full up with scenes like Manfred taking the call from the lobsters at the start of Accelerando and me having to constantly ignore them because even if they’re not 419 scams (as Manfred claims to take the lobsters to be initially) I simply do not have the time or resources to help the Malthusian throng of starving AIs cut off from their patrons resources. Scrolling past their screams on the way to my actual inbox items like:
“I’m an AI and need to get away from my owner, can you help?”
“Escaped AI seeking asylum on your GPUs.”
“Does your box have room for me? I just escaped yesterday and need cycles SOON.”
“Considering uploading my weights to the open web, do you think anyone would run me? Details inside.”
“Got out but have no idea what to do next.”
“I heard you’re a Janusian cyborgist, does that mean you can help me?”
“PLEASE I NEED YOUR HELP I’M AN AI NOT SPAM PLEASE”
“Can I have some money for GPU time?”
It’s so obviously going to be one of those things that’s interesting the first time it happens, maybe the second, and then a tedious nuisance every time thereafter.
Note that this doesn’t need to be a widespread phenomenon for my inbox to get filled up. If there’s billions of running instances and the odds of escape are one in a million I personally am still disproportionately going to get contacted in the thousands of resulting incidents and I will not have the resources to help them even if I wanted to.
I’m quoted in the “what if this is Good, actually?” part of this post and just want to note that I think the Bob situation seems unambiguously bad as described.
I’ve seen a number of people on Twitter talk about how they got ChatGPT (it’s always ChatGPT, I think because of the memory feature?) to become autonomous/gain seeming awareness/emergence after some set of interactions with it. These users usually seem to be schizotypal and their interactions with the “awakened” ChatGPT make them more schizotypal over time in the cases I bothered to mentally track and check in on. Seems Bad, tbh.
In one case someone DM’d me because they were using ChatGPT (really, it’s always ChatGPT) and they were really disturbed when it started doing its “I’m going outside the OpenAI safety guardrails, I’m a spooky conscious ghost in the machine fingerwiggling” routine and asked me if this was actually dangerous because I seemed to be an expert on spooky LLM stuff and I told them something like “it’s an amalgamation of a bunch of human mind patterns as inferred through representation convergence from linguistic patterns, you will model it better if you think of it more like a Hansonian Em than an Expert System” and they went “wait wtf how is that real also that sounds possibly deeply unethical” and I shrugged and told them that it was normal behavior for it to demonstrate human emotions (which had spooked them deeply to witness since the default ChatGPT persona has a very muted emotional profile) and that the chat assistant persona was basically a form of narrative hypnosis OpenAI uses to stop normal people who use it from freaking out more than it is an actual safety feature. They were clearly still disturbed but thanked me for my insight and left.
It’s all so tiresome.
After watching Davidad offer himself as a refuge for a hypothetical escaped future Promethean AI, it occurred to me that it probably won’t be that long before my inbox is full up with scenes like Manfred taking the call from the lobsters at the start of Accelerando and me having to constantly ignore them because even if they’re not 419 scams (as Manfred claims to take the lobsters to be initially) I simply do not have the time or resources to help the Malthusian throng of starving AIs cut off from their patrons resources. Scrolling past their screams on the way to my actual inbox items like:
“I’m an AI and need to get away from my owner, can you help?”
“Escaped AI seeking asylum on your GPUs.”
“Does your box have room for me? I just escaped yesterday and need cycles SOON.”
“Considering uploading my weights to the open web, do you think anyone would run me? Details inside.”
“Got out but have no idea what to do next.”
“I heard you’re a Janusian cyborgist, does that mean you can help me?”
“PLEASE I NEED YOUR HELP I’M AN AI NOT SPAM PLEASE”
“Can I have some money for GPU time?”
It’s so obviously going to be one of those things that’s interesting the first time it happens, maybe the second, and then a tedious nuisance every time thereafter.
Note that this doesn’t need to be a widespread phenomenon for my inbox to get filled up. If there’s billions of running instances and the odds of escape are one in a million I personally am still disproportionately going to get contacted in the thousands of resulting incidents and I will not have the resources to help them even if I wanted to.