My impression was that it was generally agreed that superintelligences sufficiently visible and slow-growing to be squelched by governments and the like aren’t much of a threat; the balrog-like (hypothetical) ones are the ones that emerge too quickly and powerfully to be so easily stopped.
I don’t think that’s the case. A superintelligence doesn’t have to be balrog like to advance to the point where it’s too big to fail and thus not easily regulated by the government.
EY et al focus more on the threat of a superintelligence that can improve itself fast and have a lot of power in short amount of time but that’s not the only concerning scenario.
When a bank like HSBC can launder drug and terrorist money without any of it’s officials going to prison for it, the amount of control that a government could exert on a big company run by a complex AI might also be quite limited.
When the superintelligence becomes good enough at making money and buying politicians, it doesn’t have to worry so much about government action, and has enough time to grow slowly.
You have at least two options: either buy Putin, or hire someone to replace him, whatever is cheaper. It’s not like Putin single-handedly rules his country—he relies on his army, police, secret services, etc. All these institutions probably have many people who would enjoy replacing Putin at the top of the pyramid. Throw in some extra money (“if you are going to replace Putin, here you have a few extra billions to bribe whoever needs to be bribed to help you with the coup”).
Or the Chinese Politbureau?
I am not familiar with the internal structure of the Chinese Politbureau, but I would guess this one is easier. There are probably competing factions, so you will support the one more friendly to you.
But there is always to option to ignore both Putin and the Chinese Politbureau, and upload yourself to a computer center built in some other country.
If you are looking at an AGI that manages investment at a company like Goldman Sachs in an effective way it doesn’t even need to know directly how to buy politicians.
If it makes a lot of money for Goldman Sachs, there are other people at Goldman who can do the job of buying politicians.
When Ray Dalio of Bridgewater Associates wants to build an AI that can replace him after he retires, it’s not clear whether any government can effectively regulate it.
I don’t think that’s the case. A superintelligence doesn’t have to be balrog like to advance to the point where it’s too big to fail and thus not easily regulated by the government.
EY et al focus more on the threat of a superintelligence that can improve itself fast and have a lot of power in short amount of time but that’s not the only concerning scenario.
When a bank like HSBC can launder drug and terrorist money without any of it’s officials going to prison for it, the amount of control that a government could exert on a big company run by a complex AI might also be quite limited.
When the superintelligence becomes good enough at making money and buying politicians, it doesn’t have to worry so much about government action, and has enough time to grow slowly.
How much does Putin cost? Or the Chinese Politbureau?
You have at least two options: either buy Putin, or hire someone to replace him, whatever is cheaper. It’s not like Putin single-handedly rules his country—he relies on his army, police, secret services, etc. All these institutions probably have many people who would enjoy replacing Putin at the top of the pyramid. Throw in some extra money (“if you are going to replace Putin, here you have a few extra billions to bribe whoever needs to be bribed to help you with the coup”).
I am not familiar with the internal structure of the Chinese Politbureau, but I would guess this one is easier. There are probably competing factions, so you will support the one more friendly to you.
But there is always to option to ignore both Putin and the Chinese Politbureau, and upload yourself to a computer center built in some other country.
Correct, and yet Putin rules with hardly a challenge to his supremacy.
Money is not very useful when you’re dead.
If you are looking at an AGI that manages investment at a company like Goldman Sachs in an effective way it doesn’t even need to know directly how to buy politicians. If it makes a lot of money for Goldman Sachs, there are other people at Goldman who can do the job of buying politicians.
When Ray Dalio of Bridgewater Associates wants to build an AI that can replace him after he retires, it’s not clear whether any government can effectively regulate it.