The key is that if AGI’s are smarter than humans those organizations run AGIs that have long-term goals will outperform organizations that mix humans with long-term goals along with AGIs that are only capable to pursue short-term goals.
One long-term goal that many AGIs are going to have is to create training data to make the AGI more effective. Training data is good when it makes the AGI more performant over a long timeframe.
AGIs that run that way are going to outperform AGIs where humans oversee all the training data.
The key is that if AGI’s are smarter than humans those organizations run AGIs that have long-term goals will outperform organizations that mix humans with long-term goals along with AGIs that are only capable to pursue short-term goals.
If the LT goal of the AI is perfectly aligned with the goals of the organisation, yes—smarter isn’t enough, it needs to be infallible. If it’s fallible, the organisation needs to be able to tweak the goals as it goes along. Remember, smarter means it’s better at executing it’s goal, not at understanding it.
The main goal of most companies is to make money. If an AGI that runs a company is better at that it outcompete other companies. It doesn’t need infallibility. Companies run by humans are also not perfectly aligned and the interests of managers and the interests of the company are different.
It’s to make money without breaking the law. An ASI that fuflfils both goals isn’t going to kill everybody, since murder is illegal. So even if you do have ASIs with stable long term goals, they don’t lead to doom. (It’s interesting to think of the chilling effect of a law that any human who creates an agentive AI is criminally responsible for what it does).
Most big companies don’t have the goal of making money without breaking the law but are often willing to break it as long as the punishment for breaking it isn’t too costly.
But even if the AGI doesn’t murder anyone in the first five year it operates it can still focus on acquiring resources and get untouchable from human actors and then engage in actions that lead to people dying. The holodomor wasn’t directly murder but people still died because they didn’t have food.
So long term goals aren’t a default; market pressure will put them there as humans slowly cede more and more control to AIs, simply because the latter are making decisions that work out better. Presumably this would start with lower level decisions (e.g. how exactly to write this line of code; which employee to reward based on performance) and then slowly be given higher level decisions to make. In particular, we don’t die the first time someone creates an AI with the ability to (escape, self improve and then) kill the competing humans, because that AI is likely focused on a much smaller more near term goal. That way, if we’re careful and clever we have a chance to study a smarter-than-human general intelligence without dying. Is that an accurate description of how you see things playing out?
The key is that if AGI’s are smarter than humans those organizations run AGIs that have long-term goals will outperform organizations that mix humans with long-term goals along with AGIs that are only capable to pursue short-term goals.
One long-term goal that many AGIs are going to have is to create training data to make the AGI more effective. Training data is good when it makes the AGI more performant over a long timeframe.
AGIs that run that way are going to outperform AGIs where humans oversee all the training data.
If the LT goal of the AI is perfectly aligned with the goals of the organisation, yes—smarter isn’t enough, it needs to be infallible. If it’s fallible, the organisation needs to be able to tweak the goals as it goes along. Remember, smarter means it’s better at executing it’s goal, not at understanding it.
The main goal of most companies is to make money. If an AGI that runs a company is better at that it outcompete other companies. It doesn’t need infallibility. Companies run by humans are also not perfectly aligned and the interests of managers and the interests of the company are different.
It’s to make money without breaking the law. An ASI that fuflfils both goals isn’t going to kill everybody, since murder is illegal. So even if you do have ASIs with stable long term goals, they don’t lead to doom. (It’s interesting to think of the chilling effect of a law that any human who creates an agentive AI is criminally responsible for what it does).
Most big companies don’t have the goal of making money without breaking the law but are often willing to break it as long as the punishment for breaking it isn’t too costly.
But even if the AGI doesn’t murder anyone in the first five year it operates it can still focus on acquiring resources and get untouchable from human actors and then engage in actions that lead to people dying. The holodomor wasn’t directly murder but people still died because they didn’t have food.
So long term goals aren’t a default; market pressure will put them there as humans slowly cede more and more control to AIs, simply because the latter are making decisions that work out better. Presumably this would start with lower level decisions (e.g. how exactly to write this line of code; which employee to reward based on performance) and then slowly be given higher level decisions to make. In particular, we don’t die the first time someone creates an AI with the ability to (escape, self improve and then) kill the competing humans, because that AI is likely focused on a much smaller more near term goal. That way, if we’re careful and clever we have a chance to study a smarter-than-human general intelligence without dying. Is that an accurate description of how you see things playing out?
If a powerful company is controlled by an AGI it doesn’t need to kill competing humans to avoid the humans from shutting the AGI down or modifying it.