If trillion-dollar tech companies stop trying to make their systems do what they want, I will update that marginal deep-thinking researchers should allocate themselves to making alignment (the scalar!) cheaper/easier/better instead of making bargaining/cooperation/mutual-governance cheaper/easier/better. I just don’t see that happening given the structure of today’s global economy and tech industry.
In your story, trillion-dollar tech companies are trying to make their systems do what they want and failing. My best understanding of your position is: “Sure, but they will be trying really hard. So additional researchers working on the problem won’t much change their probability of success, and you should instead work on more-neglected problems.”
My position is:
Eventually people will work on these problems, but right now they are not working on them very much and so a few people can be a big proportional difference.
If there is going to be a huge investment in the future, then early investment and training can effectively be very leveraged. Scaling up fields extremely quickly is really difficult for a bunch of reasons.
It seems like AI progress may be quite fast, such that it will be extra hard to solve these problems just-in-time if we don’t have any idea what we are doing in advance.
On top of all that, for many use cases people will actually be reasonably happy with misaligned systems like those in your story (that e.g. appear to be doing a good job, keep the board happy, perform well as evaluated by the best human-legible audits...). So it seems like commercial incentives may not push us to safe levels of alignment.
My best understanding of your position is: “Sure, but they will be trying really hard. So additional researchers working on the problem won’t much change their probability of success, and you should instead work on more-neglected problems.”
That is not my position if “you” in the story is “you, Paul Christiano” :) The closest position I have to that one is : “If another Paul comes along who cares about x-risk, they’ll have more positive impact by focusing on multi-agent and multi-stakeholder issues or ‘ethics’ with AI tech than if they focus on intent alignment, because multi-agent and multi-stakeholder dynamics will greatly affect what strategies AI stakeholders ‘want’ their AI systems to pursue.”
If they tried to get you to quit working on alignment, I’d say “No, the tech companies still need people working on alignment for them, and Paul is/was one of those people. I don’t endorse converting existing alignment researchers to working on multi/multi delegation theory (unless they’re naturally interested in it), but if a marginal AI-capabilities-bound researcher comes along, I endorse getting them set up to think about multi/multi delegation more than alignment.”
In your story, trillion-dollar tech companies are trying to make their systems do what they want and failing. My best understanding of your position is: “Sure, but they will be trying really hard. So additional researchers working on the problem won’t much change their probability of success, and you should instead work on more-neglected problems.”
My position is:
Eventually people will work on these problems, but right now they are not working on them very much and so a few people can be a big proportional difference.
If there is going to be a huge investment in the future, then early investment and training can effectively be very leveraged. Scaling up fields extremely quickly is really difficult for a bunch of reasons.
It seems like AI progress may be quite fast, such that it will be extra hard to solve these problems just-in-time if we don’t have any idea what we are doing in advance.
On top of all that, for many use cases people will actually be reasonably happy with misaligned systems like those in your story (that e.g. appear to be doing a good job, keep the board happy, perform well as evaluated by the best human-legible audits...). So it seems like commercial incentives may not push us to safe levels of alignment.
That is not my position if “you” in the story is “you, Paul Christiano” :) The closest position I have to that one is : “If another Paul comes along who cares about x-risk, they’ll have more positive impact by focusing on multi-agent and multi-stakeholder issues or ‘ethics’ with AI tech than if they focus on intent alignment, because multi-agent and multi-stakeholder dynamics will greatly affect what strategies AI stakeholders ‘want’ their AI systems to pursue.”
If they tried to get you to quit working on alignment, I’d say “No, the tech companies still need people working on alignment for them, and Paul is/was one of those people. I don’t endorse converting existing alignment researchers to working on multi/multi delegation theory (unless they’re naturally interested in it), but if a marginal AI-capabilities-bound researcher comes along, I endorse getting them set up to think about multi/multi delegation more than alignment.”