Sorry, but I only skimmed this… The supposition seems to be that “human with AI advisor” will always stay ahead of “pure AI”. But how easily does “human with AI advisor” turn into “AI with a human peripheral” or “posthuman with an AI exocortex”? Is there some reason why neurons are better than transistors at executive functions? This essay is like saying “the neocortex will change the world but the midbrain will still be in charge”.
I think that if humans with AI advisors are approximately as competent as pure AI in terms of pure capabilities, I would expect that humans with AI advisors would outcompete the pure AI in practice given that the humans appear more aligned and less likely to be dangerous then pure AI—a significant competitive advantage in a lot of power seeking scenarios where gaining the trust of other agents is important.
Yes, we usually select our leaders (e.g., presidents) not for their cognitive abilities but literally for how “aligned “ we believe they are with our interest. Even if we completely solve the alignment problem, AI would likely face an uphill battle in overcoming prejudice and convincing people that they are as aligned as an alternative human. As the saying goes for many discriminated groups, they would have to be twice as good to get to the same place.
We do not assume that humans are superior to AI in any way, or that neurons are superior to transistors. Similarly we do not claim that an AI CEO would be inferior to a human one. Rather we only claim that it would not dominate a human CEO as an AI chess player is to a human chess player. Note that currently, CEOs are usually not the smartest employees in their company, but that does not mean that they are the peripheral of their smartest engineers.
Sorry, but I only skimmed this… The supposition seems to be that “human with AI advisor” will always stay ahead of “pure AI”. But how easily does “human with AI advisor” turn into “AI with a human peripheral” or “posthuman with an AI exocortex”? Is there some reason why neurons are better than transistors at executive functions? This essay is like saying “the neocortex will change the world but the midbrain will still be in charge”.
I think that if humans with AI advisors are approximately as competent as pure AI in terms of pure capabilities, I would expect that humans with AI advisors would outcompete the pure AI in practice given that the humans appear more aligned and less likely to be dangerous then pure AI—a significant competitive advantage in a lot of power seeking scenarios where gaining the trust of other agents is important.
Yes, we usually select our leaders (e.g., presidents) not for their cognitive abilities but literally for how “aligned “ we believe they are with our interest. Even if we completely solve the alignment problem, AI would likely face an uphill battle in overcoming prejudice and convincing people that they are as aligned as an alternative human. As the saying goes for many discriminated groups, they would have to be twice as good to get to the same place.
We do not assume that humans are superior to AI in any way, or that neurons are superior to transistors. Similarly we do not claim that an AI CEO would be inferior to a human one. Rather we only claim that it would not dominate a human CEO as an AI chess player is to a human chess player. Note that currently, CEOs are usually not the smartest employees in their company, but that does not mean that they are the peripheral of their smartest engineers.