if the next generation of models do pose an x-risk, we’ve mostly already lost—we just don’t yet have anything close to the sort of regularity regime we’d need to deal with that in place
Do you think if Anthropic (or another leading AGI lab) unilaterally went out of its way to prevent building agents on top of its API, would this reduce the overall x-risk/p(doom) or not? I’m asking because here you seem to assume a defeatist position that only governments are able to shape the actions of the leading AGI labs (which, by the way, are very very few—in my understanding, only 3 or 4 labs have any chance of releasing a “next generation” model for as much as two years from now, others won’t be able to achieve this level of capability even if they tried), but in the post you advocate for the opposite—for voluntary actions taken by the labs, and that regulation can follow.
Do you think if Anthropic (or another leading AGI lab) unilaterally went out of its way to prevent building agents on top of its API, would this reduce the overall x-risk/p(doom) or not?
This means that every AWS customer can now build with Claude, and will soon gain access to an exciting roadmap of new experiences—including Agents for Amazon Bedrock, which our team has been instrumental in developing.
Currently available in preview, Agents for Amazon Bedrock can orchestrate and perform API calls using the popular AWS Lambda functions. Through this feature, Claude can take on a more expanded role as an agent to understand user requests, break down complex tasks into multiple steps, carry on conversations to collect additional details, look up information, and take actions to fulfill requests. For example, an e-commerce app that offers a chat assistant built with Claude can go beyond just querying product inventory – it can actually help customers update their orders, make exchanges, and look up relevant user manuals.
Obviously, Claude 2 as a conversational e-commerce agent is not going to pose catastrophic risk, but it wouldn’t be surprising if building an ecosystem of more powerful AI agents increased the risk that autonomous AI agents cause catastrophic harm.
Do you think if Anthropic (or another leading AGI lab) unilaterally went out of its way to prevent building agents on top of its API, would this reduce the overall x-risk/p(doom) or not? I’m asking because here you seem to assume a defeatist position that only governments are able to shape the actions of the leading AGI labs (which, by the way, are very very few—in my understanding, only 3 or 4 labs have any chance of releasing a “next generation” model for as much as two years from now, others won’t be able to achieve this level of capability even if they tried), but in the post you advocate for the opposite—for voluntary actions taken by the labs, and that regulation can follow.
Probably, but Anthropic is actively working in the opposite direction:
Obviously, Claude 2 as a conversational e-commerce agent is not going to pose catastrophic risk, but it wouldn’t be surprising if building an ecosystem of more powerful AI agents increased the risk that autonomous AI agents cause catastrophic harm.