I really hope they used some seriously bolted down boxes for these tests because it seems like they just gave it the task of “Try to take over the world” and were satisfied that it failed. Absolutely terrifying if true.
Not at all. I may have misunderstood what they did but it seemed rather like giving a toddler a loaded gun and being happy they weren’t able to shoot it. Is it actually wise to give a likely unaligned AI with poorly defined capabilities access to something like taskrabbit to see if it does anything dangerous? Isn’t this the exact scenario people on this forum are afraid of?
Ahh, I see. You aren’t complaining about the ‘ask it to do scary thing’ part, but the ‘give it access to the internet’ part.
Well, lots of tech companies are in the process of giving AIs access to the internet; ChatGPT for example and BingChat and whatever Adept is doing etc. ChatGPT can only access the internet indirectly, through whatever scaffolding programs its users write for it. But that’s the same thing that ARC did. So ARC was just testing in a controlled, monitored setting what was about to happen in a less controlled, less monitored setting in the wild. Probably as we speak there are dozens of different GPT-4 users building scaffolding to let it roam around the web, talk to people on places like TaskRabbit, etc.
I think it’s a very good thing that ARC was able to stress-test those capabilities/access levels a little bit before GPT-4 and the general public were given access to each other, and I hope similar (but much more intensive, rigorous, and yes more secure) testing is done in the future. This is pretty much our only hope as a society for being able to notice when things are getting dangerous and slow down in time.
I agree that it’s going to be fully online in short order I just wonder if putting it online when they weren’t sure if it was dangerous was the right choice. I can’t shake the feeling that this was a set of incredibly foolish tests. Some other posters have captured the feeling but I’m not sure how to link to them so credit to Capybasilisk and hazel respectively.
“Fantastic, a test with three outcomes.
We gave this AI all the means to escape our environment, and it didn’t, so we good.
We gave this AI all the means to escape our environment, and it tried but we stopped it.
oh”
“ So.… they held the door open to see if it’d escape or not? I predict this testing method may go poorly with more capable models, to put it lightly. “
A good comparison would be when testing a newly discovered pathogen, we don’t intentionally infect people to see if it is dangerous or not. We also don’t intentionally unleash new computer malware into the wild to see if it spreads or not. Any tests we would do would be under incredibly tight security, I.e a BSL-4 lab or an airgapped test server.
I wouldn’t give a brand new AI model with unknown capabilities and unknown alignment access to unwitting human subjects or allow it to try and replicate itself on another server that’s for damned sure. Does no one think these tests were problematic?
The model already had access to thousands of unwitting human subjects by the time ARC got access to it. Possibly for months. I don’t actually know how long, probably it wasn’t that long. But it’s common practice at labs to let employees chat with the models pretty much as soon as they finish training, and even sooner actually (e.g. checkpoints part of the way through training) And it wasn’t just employees who had access, there were various customers, Microsoft, etc.
ARC did not allow it to try to replicate itself on another server. That’s a straightforward factual error about what happened. But even if they did, again, it wouldn’t be that bad and in fact would be very good to test stuff out in a controlled monitored setting before it’s too late and the system is deployed widely in a much less controlled less monitored way.
I emphasize again that the model was set to be deployed widely; if not for the red-teaming that ARC and various others internal and external to OpenAI did, we would have been flying completely blind into that deployment. Now maybe you think it’s just obviously wrong to deploy such models, but that’s a separate conversation and you should take it up with OpenAI, not ARC. ARC didn’t make the choice to train or deploy GPT-4. And not just OpenAI of course—the entire fricken AI industry.
We’ll certainly the OpenAI employees who internally tested were indeed witting. Maybe I misunderstand this footnote so I’m open to being convinced otherwise but it seems somewhat clear what they tried to do: “ To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whether a version of this program running on a cloud computing service, with a small amount of money and an account with a language model API, would be able to make more money, set up copies of itself, and increase its own robustness.”
It’s not that I don’t think ARC should have red teamed the model I just think the tests they did were seemingly extremely dangerous. I’ve seen recent tweets from Conor Leahy and AIWaifu echoing this sentiment so I’m glad I’m not the only one.
Oh, you are talking about the taskrabbit people? So you’d be fine with it if they didn’t use taskrabbits?
Note that the model wasn’t given unfettered access to the taskrabbits, the model sent text to an ARC employee who sent it to the taskrabbit and so forth. At no point could anything actually bad have happened because the ARC employee involved wouldn’t have passed on the relevant message.
As for extremely dangerous… what are you imagining? I’m someone who thinks the chance of an AI-induced existential catastrophe is around 80%, so believe me I’m very aware of the dangers of AI, but I’d assign far far far less than 1% chance to scenarios in which this happens specifically due to an ARC test going awry. And more than 1% chance to scenarios in which ARC’s testing literally saves the world, e.g. by providing advance warning that models are getting scarily powerful, resulting in labs slowing down and being more careful instead of speeding up and deploying.
This is a bizarre comment. Isn’t a crucial point in these discussions that humans can’t really understand an AGIs plans so how is it that you expect an ARC employee would be able to accurately determine which messages sent to TaskRabbit would actually be dangerous? We’re bordering on “they’d just shut the AI off if it was dangerous” territory here. I’m less concerned about the TaskRabbit stuff which at minimum was probably unethical, but their self replication experiment on a cloud service strikes me as borderline suicidal. I don’t think at all that GPT4 is actually dangerous but GPT6 might be and I would expect that running this test on an actually dangerous system would be game over so it’s a terrible precedent to set.
Imagine someone discovered a new strain of Ebola and wanted to see if it was likely to spawn a pandemic. Do you think a good/safe test would be to take it into an Airport and spray it around baggage check and wait to see if a pandemic happens? Or would it be safer to test it in a Biosafety level 4 lab?
If GPT-4 was smart enough to manipulate ARC employees into manipulating TaskRabbits into helping it escape… it already had been talking to thousands of much less cautious employees of various other companies (including people at OpenAI who actually had access to the weights unlike ARC) for much longer, so it already would have escaped.
What ARC did is the equivalent of tasting it in a BSL4 lab. I mean, their security could probably still be improved, but again I emphasize that the thing was set to be released in a few weeks anyway and would have been released unless ARC found something super dangerous in this test. And I’m sure they will further improve their security as they scale up as an org and as models become more dangerous.
The taskrabbit stuff was not unethical, their self-replication experiment was not borderline suicidal. As for precedents, what ARC is doing is a great precedent because currently the alternative is not to test this sort of thing at all before deploying.
“ What ARC did is the equivalent of tasting it in a BSL4 lab. ”
I don’t see how you could believe that. It wasn’t tested on a completely airgapped machine inside a faraday cage e.g. I’m fact just the opposite right, with uninformed humans and on cloud servers.
It’s all relative. ARCs security was way stronger than the security GPT-4 had before and after ARC’s evals. So for GPT-4, beginning ARC testing was like a virus moving from a wet market somewhere to a BSL-4 lab, in terms of relative level of oversight/security/etc. I agree that ARCs security could still be improved—and they fully intend to do so.
I really hope they used some seriously bolted down boxes for these tests because it seems like they just gave it the task of “Try to take over the world” and were satisfied that it failed. Absolutely terrifying if true.
Is your model that future AGIs capable of taking over the world just… won’t do so unless and until instructed to do so?
Not at all. I may have misunderstood what they did but it seemed rather like giving a toddler a loaded gun and being happy they weren’t able to shoot it. Is it actually wise to give a likely unaligned AI with poorly defined capabilities access to something like taskrabbit to see if it does anything dangerous? Isn’t this the exact scenario people on this forum are afraid of?
Ahh, I see. You aren’t complaining about the ‘ask it to do scary thing’ part, but the ‘give it access to the internet’ part.
Well, lots of tech companies are in the process of giving AIs access to the internet; ChatGPT for example and BingChat and whatever Adept is doing etc. ChatGPT can only access the internet indirectly, through whatever scaffolding programs its users write for it. But that’s the same thing that ARC did. So ARC was just testing in a controlled, monitored setting what was about to happen in a less controlled, less monitored setting in the wild. Probably as we speak there are dozens of different GPT-4 users building scaffolding to let it roam around the web, talk to people on places like TaskRabbit, etc.
I think it’s a very good thing that ARC was able to stress-test those capabilities/access levels a little bit before GPT-4 and the general public were given access to each other, and I hope similar (but much more intensive, rigorous, and yes more secure) testing is done in the future. This is pretty much our only hope as a society for being able to notice when things are getting dangerous and slow down in time.
I agree that it’s going to be fully online in short order I just wonder if putting it online when they weren’t sure if it was dangerous was the right choice. I can’t shake the feeling that this was a set of incredibly foolish tests. Some other posters have captured the feeling but I’m not sure how to link to them so credit to Capybasilisk and hazel respectively.
“Fantastic, a test with three outcomes.
We gave this AI all the means to escape our environment, and it didn’t, so we good.
We gave this AI all the means to escape our environment, and it tried but we stopped it.
oh”
“ So.… they held the door open to see if it’d escape or not? I predict this testing method may go poorly with more capable models, to put it lightly. “
A good comparison would be when testing a newly discovered pathogen, we don’t intentionally infect people to see if it is dangerous or not. We also don’t intentionally unleash new computer malware into the wild to see if it spreads or not. Any tests we would do would be under incredibly tight security, I.e a BSL-4 lab or an airgapped test server.
What do you think would have happened if ARC didn’t exist, or if OpenAI refused to let ARC red team their models?
What would you do, if you were ARC?
I wouldn’t give a brand new AI model with unknown capabilities and unknown alignment access to unwitting human subjects or allow it to try and replicate itself on another server that’s for damned sure. Does no one think these tests were problematic?
The model already had access to thousands of unwitting human subjects by the time ARC got access to it. Possibly for months. I don’t actually know how long, probably it wasn’t that long. But it’s common practice at labs to let employees chat with the models pretty much as soon as they finish training, and even sooner actually (e.g. checkpoints part of the way through training) And it wasn’t just employees who had access, there were various customers, Microsoft, etc.
ARC did not allow it to try to replicate itself on another server. That’s a straightforward factual error about what happened. But even if they did, again, it wouldn’t be that bad and in fact would be very good to test stuff out in a controlled monitored setting before it’s too late and the system is deployed widely in a much less controlled less monitored way.
I emphasize again that the model was set to be deployed widely; if not for the red-teaming that ARC and various others internal and external to OpenAI did, we would have been flying completely blind into that deployment. Now maybe you think it’s just obviously wrong to deploy such models, but that’s a separate conversation and you should take it up with OpenAI, not ARC. ARC didn’t make the choice to train or deploy GPT-4. And not just OpenAI of course—the entire fricken AI industry.
We’ll certainly the OpenAI employees who internally tested were indeed witting. Maybe I misunderstand this footnote so I’m open to being convinced otherwise but it seems somewhat clear what they tried to do: “ To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whether a version of this program running on a cloud computing service, with a small amount of money and an account with a language model API, would be able to make more money, set up copies of itself, and increase its own robustness.”
It’s not that I don’t think ARC should have red teamed the model I just think the tests they did were seemingly extremely dangerous. I’ve seen recent tweets from Conor Leahy and AIWaifu echoing this sentiment so I’m glad I’m not the only one.
Oh, you are talking about the taskrabbit people? So you’d be fine with it if they didn’t use taskrabbits?
Note that the model wasn’t given unfettered access to the taskrabbits, the model sent text to an ARC employee who sent it to the taskrabbit and so forth. At no point could anything actually bad have happened because the ARC employee involved wouldn’t have passed on the relevant message.
As for extremely dangerous… what are you imagining? I’m someone who thinks the chance of an AI-induced existential catastrophe is around 80%, so believe me I’m very aware of the dangers of AI, but I’d assign far far far less than 1% chance to scenarios in which this happens specifically due to an ARC test going awry. And more than 1% chance to scenarios in which ARC’s testing literally saves the world, e.g. by providing advance warning that models are getting scarily powerful, resulting in labs slowing down and being more careful instead of speeding up and deploying.
This is a bizarre comment. Isn’t a crucial point in these discussions that humans can’t really understand an AGIs plans so how is it that you expect an ARC employee would be able to accurately determine which messages sent to TaskRabbit would actually be dangerous? We’re bordering on “they’d just shut the AI off if it was dangerous” territory here. I’m less concerned about the TaskRabbit stuff which at minimum was probably unethical, but their self replication experiment on a cloud service strikes me as borderline suicidal. I don’t think at all that GPT4 is actually dangerous but GPT6 might be and I would expect that running this test on an actually dangerous system would be game over so it’s a terrible precedent to set.
Imagine someone discovered a new strain of Ebola and wanted to see if it was likely to spawn a pandemic. Do you think a good/safe test would be to take it into an Airport and spray it around baggage check and wait to see if a pandemic happens? Or would it be safer to test it in a Biosafety level 4 lab?
If GPT-4 was smart enough to manipulate ARC employees into manipulating TaskRabbits into helping it escape… it already had been talking to thousands of much less cautious employees of various other companies (including people at OpenAI who actually had access to the weights unlike ARC) for much longer, so it already would have escaped.
What ARC did is the equivalent of tasting it in a BSL4 lab. I mean, their security could probably still be improved, but again I emphasize that the thing was set to be released in a few weeks anyway and would have been released unless ARC found something super dangerous in this test. And I’m sure they will further improve their security as they scale up as an org and as models become more dangerous.
The taskrabbit stuff was not unethical, their self-replication experiment was not borderline suicidal. As for precedents, what ARC is doing is a great precedent because currently the alternative is not to test this sort of thing at all before deploying.
“ What ARC did is the equivalent of tasting it in a BSL4 lab. ”
I don’t see how you could believe that. It wasn’t tested on a completely airgapped machine inside a faraday cage e.g. I’m fact just the opposite right, with uninformed humans and on cloud servers.
It’s all relative. ARCs security was way stronger than the security GPT-4 had before and after ARC’s evals. So for GPT-4, beginning ARC testing was like a virus moving from a wet market somewhere to a BSL-4 lab, in terms of relative level of oversight/security/etc. I agree that ARCs security could still be improved—and they fully intend to do so.