Thanks for your comments. How do you think human intelligence works? Perhaps by doing a massive parallel search to approximate the best solution?
The brain architecture is highly parallel, however, how it forms high-level thoughts is not known. My guess is that’s some sort of parallel Monte Carlo search driven by complex, partially innate and partially learned, heuristics.
m confused… if time required is a polynomial or exponential function of your problem size, then hardware that runs twice as fast will still solve your problem twice as fast, won’t it?
Yes, but it wouldn’t be twice as smart. If you were to speed up a chicken brain by a factor of 10,000 you wouldn’t get a super-human intelligence.
And if the algorithm you’re using to solve the problem is perfectly parallelizable (which I grant to AI foom proponents ’cause it seems plausible to me)
Perfect parallelizability (linear speedup in the number of processors) is physically impossible due to the fact that information propagates at finite speed, though depending on hardware details, as long as your computer doesn’t get too big, you can obtain close to linear speedups on certain problems. NP-complete problems can be solved by brute-force exhaustive search, in principle, which is highly parallelizable. But exhaustive search has a very fast growing exponential complexity, hence it doesn’t get you very far from toy problem instances even on parallel hardware. The more complex heuristics and approximation schemes you use, the less parallelizability you get, in general.
Anyway, 10,000 chickens won’t make a super-human intelligence, even if you found some way to wire their brains togheter.
If you were to speed up a chicken brain by a factor of 10,000 you wouldn’t get a super-human intelligence.
Sure, but if we assume we manage to have a human-level AI, how powerful should we expect it to be if we speed that up by a factor of 10, 100, or more?
Personally, I’m pretty sure such a thing is still powerful enough to take over the world (assuming it is the only such AI), and in any case dangerous enough to lock us all in a future we really don’t want.
At that point, I don’t really care if it’s “superhuman” or not.
It won’t be any smarter at all actually, it will just have more relative time.
Basically, if you take someone, and give them 100 days to do something, they will have 100 times as much time to do it as they would if it takes 1 day, but if it is beyond their capabilities, then it will remain beyond their capabilities, and running at 100x speed is only helpful for projects for which mental time is the major factor—if you have to run experiments and wait for results, all you’re really doing is decreasing the lag time between experiments, and even then only potentially.
Its not even as good as having 100 slaves work on a project (as someone else posited) because you’re really just having ONE slave work on the project for 100 days; copying them 100 times likely won’t help that issue.
This is one of the fundamental problems with the idea of the singularity in the first place; the truth is that designing more intelligent intelligences is probably HARDER than designing simpler ones, possibly by orders of magnitude, and it may not be scalar at all. If you look at rodent brains and human brains, there are numerous differences between them—scaling up a rodent brain to the same EQ as a human brain would NOT give you something as smart as a human, or even sapient.
You are very likely to see declining returns, not accelerating returns, which is exactly what we see in all other fields of technology—the higher you get, the harder it is to go further.
Moreover, it isn’t even clear what a “superhuman” intelligence even means. We don’t even have any way of measuring intelligence absolutely that I am aware of—IQ is a statistical means, as are standardized tests. We can’t say that human A is twice as smart as human B, and without such metrication it may be difficult to determine just how much smarter anything is than a human in the first place. If four geniuses can work together and get the same result as a computer which takes 1000 times as much energy to do the same task, then the computer is inefficient no matter how smart it is.
This efficiency is ANOTHER major barrier as well—human brains run off of cherrios, whereas any AI we build is going to be massively less efficient in terms of energy usage per cycle, at least for the foreseeable futures.
Another question is whether there is some sort of effective cap to intelligence given energy, heat dissipation, proximity of processing centers, ect. Given that we’re only going to see microchips 256 times as dense on a plane as we have presently available, and given the various issues with heat dissipation of 3D chips (not to mention expense), we may well run into some barriers here.
I was looking at some stuff last night and while people claim we may be able to model the brain using an exascale computer, I am actually rather skeptical after reading up on it—while 150 trillion connections between 86 billion neurons doesn’t sound like that much on the exascale, we have a lot of other things, such as glial cells, which appear to play a role in intelligence, and it is not unlikely that their function is completely vital in a proper simulation. Indeed, our utter lack of understanding of how the human brain works is a major barrier for even thinking about how we can make something more intelligent than a human which is not a human—its pretty much pure fantasy at this point. It may be that ridiculous parallelization with low latency is absolutely vital for sapience, and that could very well put a major crimp on silicon-based intelligences at all, due to their more linear nature, even with things like GPUs and multicore processors because the human brain is sending out trillions of signals with each step.
Some possibilities for simulating the human brain could easily take 10^22 FLOPS or more, and given the limitations of transistor-based computing, that looks like it is about the level of supercomputer we’d have in 2030 or so—but I wouldn’t expect much better than that beyond that point because the only way to make better processors at that point is going up or out, and to what extent we can continue doing that… well, we’ll have to see, but it would very likely eat up even more power and I would have to question the ROI at some point. We DO need to figure out how intelligence works, if only because it might make enhancing humans easier—indeed, unless intelligence is highly computationally efficient, organic intelligences may well be the optimal solution from the standpoint of efficiency, and no sort of exponential takeoff is really possible, or even likely, with such.
You are very likely to see declining returns, not accelerating returns, which is exactly what we see in all other fields of technology—the higher you get, the harder it is to go further.
In many fields of technology, we see sigmoid curves, where initial advancements lead to accelerating returns until it becomes difficult to move further ahead without running up against hard problems or fundamental limits, and returns diminish.
Making an artificial intelligence as capable as a human intelligence may be difficult, but that doesn’t mean that if we reach that point, we’ll be facing major barriers to further progression. I would say we don’t have much evidence to suggest humans are even near the ceiling of what’s strictly possible with a purely biological intelligence; we’ve had very little opportunity for further biological development since the point when cultural developments started accounting for most of our environmental viability, plus we face engineering challenges such as only being able to shove so large a cranium through a bipedal pelvis.
We have no way to even measure intelligence, let alone determine how close to capacity we’re at. We could be 90% there, or 1%, and we have no way, presently, of distinguishing between the two.
We are the smartest creatures ever to have lived on the planet Earth as far as we can tell, and given that we have seen no signs of extraterrestrial civilization, we could very well be the most intelligent creatures in the galaxy for all we know.
As for shoving out humans, isn’t the simplest solution to that simply growing them in artificial wombs?
We already have a simpler solution than that, namely the Cesarian section. It hasn’t been a safe option long enough to have had a significant impact as an evolutionary force though. Plus, there hasn’t been a lot of evolutionary pressure for increased intelligence since the advent of agriculture.
We might be the most intelligent creatures in the galaxy, but that’s a very different matter from being near the most intelligent things that could be constructed out of a comparable amount of matter. Natural selection isn’t that great a process for optimizing intelligence, it’s backpedaled on hominids before given the right niche to fill, so while we don’t have a process for measuring how close we are to the ceiling, I think the reasonable prior on our being close to it is pretty low.
Sure, but if we assume we manage to have a human-level AI, how powerful should we expect it to be if we speed that up by a factor of 10, 100, or more?
As powerful as a a team of 10, 100 human slaves, or a little more, but within the same order or magnitude.
Personally, I’m pretty sure such a think is still powerful enough to take over the world (assuming it is the only such AI), and in any case dangerous enough to lock us all in a future we really don’t want.
At first. If the “100 slaves” AI ever gets out of the box, you can multiply the initial number by the amount of hardware it can copy itself to. It can hack computers, earn (or steal) money, buy hardware…
And suddenly we’re talking about a highly coordinated team of millions.
That’s the plot of the Terminator movies, but it doesn’t seem to be a likely scenario.
During their regime, the Nazis locked up, used as slave labor, and eventually killed, millions of people. Most of them were Ashkenazi Jews, perhaps the smartest of all ethnic groups, with a language difficult to comprehend to outsiders, living in close-knit communities with transnational range, and strong inter-community ties. Did they get “out of the box” and take over the Third Reich? Nope.
AIs might have some advantages for being digital, but also disadvantages.
I think you miss the part where the team of millions continues its self-copying until it eats up every available computing power. If there’s any significant computing overhang, the AI could easily seize control of way more computing power than all the human brains put together.
Also, I think you underestimate the “highly coordinated” part. Any copy of the AI will likely share the exact same goals, and the exact same beliefs. Its instances will have common knowledge of this fact. This would creates an unprecedented level of trust. (The only possible exception I can think of are twins. And even so…)
So, let’s recap:
Thinks 100 times faster than a human, though no better.
Can copy itself over many times (the exact amount depends on computing power available).
The resulting team forms a nearly perfectly coordinated group.
Do you at least concede that this is potentially more dangerous than a whole country armed up with nukes? Would you rely on it being less dangerous than that?
When I imagine that I could make my copy which would be identical to me, sharing my goals, able to copy its experiences back to me, and willing to die for me (something like Naruto’s clones), taking over the society seems rather easy. (Assuming that no one else has this ability, and no one suspects me of having it. In real life it would probably help if all the clones looked different, but had an ability to recognize each other.)
Research: For each interesting topic I could make dozen clones which would study the topic in libraries and universities, and discuss their findings with each other. I don’t suppose it would make me an expert on everything, but I could get at least all the university-level education on most things. Resources: If I can make more money than I spend, and if I don’t make too much copies to imbalance the economy, I can let a few dozen clones work and produce the money for the rest of them. At least in the starting phase, until my research groups discover better ways to make money. Contacts: Different clones could go to different places making useful contacts wil different kinds of people. Sometimes you find a person which can help your goals significantly. With many clones I could make contacts in many different social groups, and overcome language or religious barriers (I can have a few clones learn the language or join the religion). Multiple “first impressions”: If I need a help of a given person or organization, I could in many cases gain their trust by sending multiple different clones to them, using different strategies to befriend them, until I find one that works. Taking over democratic organizations: Any organization with low barriers to entry and democratic voting can be taken over by sending enough clones there, and then voting some of the clones as new leaders. A typical non-governmental organization or even a smaller political party could be gained this way. I don’t even need a majority of clones there: two potential leaders competing with each other, half dozen experts openly supporting each of them, and dozen people befriending random voters and explaining them why leader X or leader Y is the perfect choice; then most of the voting would be done by other people. Assassination: If someone is too much of a problem, I can create a clone which kills them and then disappears. This should be used very rarely, not to draw attention to my abilities. Safety: To protect myself, I would send my different clones to different countries over the world. Joining all the winning sides: If there is an important group of people, I could join them, even the groups fighting against each other. Whoever wins, some of my clones are on the winning side.
If that AI runs on expensive or specialized hardware, it can’t necessarily expand much. For instance, if it runs on hardware worth millions of dollars, it can’t exactly copy itself just anywhere yet. Assuming that the first AI of that level will be cutting edge research and won’t be cheap, that gives a certain time window to study it safely.
The AI may be dangerous if it appeared now, but if it appears in, say, fifty years, then it will have to deal with the state of the art fifty years from now. Expanding without getting caught might be considerably more difficult then than it is now—weak AI will be all over the place, for one.
Last, but not least, the AI must have access to its own source code in order to copy it. That’s far from a given, especially if it’s a neural architecture. A human-level AI would not know how it works any more than we know how we work, so if it has no read access to itself or no way to probe its own circuitry, it won’t be able to copy itself at all. I doubt the first AI would actually have fine-grained access to its own inner workings, and I doubt it would have anywhere close to the amount of resources required to reverse engineer itself. Of course, that point is moot if some fool does give it access...
I agree with your first point, though it gets worse for us as hardware gets cheaper and cheaper.
I like your second point even more: it’s actionable. We could work on the security of personal computers.
That last one is incorrect however. The AI only have to access its object code in order to copy itself. That’s something even current computer viruses can do. And we’re back to boxing it.
If the AI is a learning system such as a neural network, and I believe that’s quite likely to be the case, there is no source/object dichotomy at all and the code may very well be unreadable outside of simple local update procedures that are completely out of the AI’s control. In other words, it might be physically impossible for both the AI and ourselves to access the AI’s object code—it would be locked in a hardware box with no physical wires to probe its contents, basically.
I mean, think of a physical hardware circuit implementing a kind of neuron network—in order for the network to be “copiable”, you need to be able to read the values of all neurons. However, that requires a global clock (to ensure synchronization, though AI might tolerate being a bit out of phase) and a large number of extra wires connecting each component to busses going out of the system. Of course, all that extra fluff inflates the cost of the system, makes it bigger, slower and probably less energy efficient. Since the first human-level AI won’t just come out of nowhere, it will probably use off-the-shelf digital neural components, and for cost and speed reasons, these components might not actually offer any way to copy their contents.
This being said, even if the AI runs on conventional hardware, locking it out of its own object code isn’t exactly rocket science. The specification of some programming languages already guarantee that this cannot happen, and type/proof theory is an active research field that may very well be able to prove the conformance of implementation to specification. If the AI is a neural network emulated on conventional hardware, the risks that it can read itself without permission are basically zilch.
There are various notions of intelligence, social intelligence includes the skills for getting in charge. My point is that human-level intelligence, even replicated or sped up, is generally not enough.
The brain architecture is highly parallel, however, how it forms high-level thoughts is not known.
My guess is that’s some sort of parallel Monte Carlo search driven by complex, partially innate and partially learned, heuristics.
Yes, but it wouldn’t be twice as smart. If you were to speed up a chicken brain by a factor of 10,000 you wouldn’t get a super-human intelligence.
Perfect parallelizability (linear speedup in the number of processors) is physically impossible due to the fact that information propagates at finite speed, though depending on hardware details, as long as your computer doesn’t get too big, you can obtain close to linear speedups on certain problems.
NP-complete problems can be solved by brute-force exhaustive search, in principle, which is highly parallelizable. But exhaustive search has a very fast growing exponential complexity, hence it doesn’t get you very far from toy problem instances even on parallel hardware. The more complex heuristics and approximation schemes you use, the less parallelizability you get, in general.
Anyway, 10,000 chickens won’t make a super-human intelligence, even if you found some way to wire their brains togheter.
One of the cooler papers I’ve seen connecting MC with thinking is http://www.stanford.edu/~ngoodman/papers/LiederGriffithsGoodman2012NIPS.pdf which claims that MCMC can even explain some cognitive biases. (I don’t know as much about MCMC as I would like, so I can’t evaluate it.)
Sure, but if we assume we manage to have a human-level AI, how powerful should we expect it to be if we speed that up by a factor of 10, 100, or more?
Personally, I’m pretty sure such a thing is still powerful enough to take over the world (assuming it is the only such AI), and in any case dangerous enough to lock us all in a future we really don’t want.
At that point, I don’t really care if it’s “superhuman” or not.
It won’t be any smarter at all actually, it will just have more relative time.
Basically, if you take someone, and give them 100 days to do something, they will have 100 times as much time to do it as they would if it takes 1 day, but if it is beyond their capabilities, then it will remain beyond their capabilities, and running at 100x speed is only helpful for projects for which mental time is the major factor—if you have to run experiments and wait for results, all you’re really doing is decreasing the lag time between experiments, and even then only potentially.
Its not even as good as having 100 slaves work on a project (as someone else posited) because you’re really just having ONE slave work on the project for 100 days; copying them 100 times likely won’t help that issue.
This is one of the fundamental problems with the idea of the singularity in the first place; the truth is that designing more intelligent intelligences is probably HARDER than designing simpler ones, possibly by orders of magnitude, and it may not be scalar at all. If you look at rodent brains and human brains, there are numerous differences between them—scaling up a rodent brain to the same EQ as a human brain would NOT give you something as smart as a human, or even sapient.
You are very likely to see declining returns, not accelerating returns, which is exactly what we see in all other fields of technology—the higher you get, the harder it is to go further.
Moreover, it isn’t even clear what a “superhuman” intelligence even means. We don’t even have any way of measuring intelligence absolutely that I am aware of—IQ is a statistical means, as are standardized tests. We can’t say that human A is twice as smart as human B, and without such metrication it may be difficult to determine just how much smarter anything is than a human in the first place. If four geniuses can work together and get the same result as a computer which takes 1000 times as much energy to do the same task, then the computer is inefficient no matter how smart it is.
This efficiency is ANOTHER major barrier as well—human brains run off of cherrios, whereas any AI we build is going to be massively less efficient in terms of energy usage per cycle, at least for the foreseeable futures.
Another question is whether there is some sort of effective cap to intelligence given energy, heat dissipation, proximity of processing centers, ect. Given that we’re only going to see microchips 256 times as dense on a plane as we have presently available, and given the various issues with heat dissipation of 3D chips (not to mention expense), we may well run into some barriers here.
I was looking at some stuff last night and while people claim we may be able to model the brain using an exascale computer, I am actually rather skeptical after reading up on it—while 150 trillion connections between 86 billion neurons doesn’t sound like that much on the exascale, we have a lot of other things, such as glial cells, which appear to play a role in intelligence, and it is not unlikely that their function is completely vital in a proper simulation. Indeed, our utter lack of understanding of how the human brain works is a major barrier for even thinking about how we can make something more intelligent than a human which is not a human—its pretty much pure fantasy at this point. It may be that ridiculous parallelization with low latency is absolutely vital for sapience, and that could very well put a major crimp on silicon-based intelligences at all, due to their more linear nature, even with things like GPUs and multicore processors because the human brain is sending out trillions of signals with each step.
Some possibilities for simulating the human brain could easily take 10^22 FLOPS or more, and given the limitations of transistor-based computing, that looks like it is about the level of supercomputer we’d have in 2030 or so—but I wouldn’t expect much better than that beyond that point because the only way to make better processors at that point is going up or out, and to what extent we can continue doing that… well, we’ll have to see, but it would very likely eat up even more power and I would have to question the ROI at some point. We DO need to figure out how intelligence works, if only because it might make enhancing humans easier—indeed, unless intelligence is highly computationally efficient, organic intelligences may well be the optimal solution from the standpoint of efficiency, and no sort of exponential takeoff is really possible, or even likely, with such.
In many fields of technology, we see sigmoid curves, where initial advancements lead to accelerating returns until it becomes difficult to move further ahead without running up against hard problems or fundamental limits, and returns diminish.
Making an artificial intelligence as capable as a human intelligence may be difficult, but that doesn’t mean that if we reach that point, we’ll be facing major barriers to further progression. I would say we don’t have much evidence to suggest humans are even near the ceiling of what’s strictly possible with a purely biological intelligence; we’ve had very little opportunity for further biological development since the point when cultural developments started accounting for most of our environmental viability, plus we face engineering challenges such as only being able to shove so large a cranium through a bipedal pelvis.
We have no way to even measure intelligence, let alone determine how close to capacity we’re at. We could be 90% there, or 1%, and we have no way, presently, of distinguishing between the two.
We are the smartest creatures ever to have lived on the planet Earth as far as we can tell, and given that we have seen no signs of extraterrestrial civilization, we could very well be the most intelligent creatures in the galaxy for all we know.
As for shoving out humans, isn’t the simplest solution to that simply growing them in artificial wombs?
We already have a simpler solution than that, namely the Cesarian section. It hasn’t been a safe option long enough to have had a significant impact as an evolutionary force though. Plus, there hasn’t been a lot of evolutionary pressure for increased intelligence since the advent of agriculture.
We might be the most intelligent creatures in the galaxy, but that’s a very different matter from being near the most intelligent things that could be constructed out of a comparable amount of matter. Natural selection isn’t that great a process for optimizing intelligence, it’s backpedaled on hominids before given the right niche to fill, so while we don’t have a process for measuring how close we are to the ceiling, I think the reasonable prior on our being close to it is pretty low.
As powerful as a a team of 10, 100 human slaves, or a little more, but within the same order or magnitude.
100 slaves are not going to take over the world.
One 10,000 year old human might be able to do it, though.
Without any legal protection?
At first. If the “100 slaves” AI ever gets out of the box, you can multiply the initial number by the amount of hardware it can copy itself to. It can hack computers, earn (or steal) money, buy hardware…
And suddenly we’re talking about a highly coordinated team of millions.
That’s the plot of the Terminator movies, but it doesn’t seem to be a likely scenario.
During their regime, the Nazis locked up, used as slave labor, and eventually killed, millions of people. Most of them were Ashkenazi Jews, perhaps the smartest of all ethnic groups, with a language difficult to comprehend to outsiders, living in close-knit communities with transnational range, and strong inter-community ties.
Did they get “out of the box” and take over the Third Reich? Nope.
AIs might have some advantages for being digital, but also disadvantages.
I think you miss the part where the team of millions continues its self-copying until it eats up every available computing power. If there’s any significant computing overhang, the AI could easily seize control of way more computing power than all the human brains put together.
Also, I think you underestimate the “highly coordinated” part. Any copy of the AI will likely share the exact same goals, and the exact same beliefs. Its instances will have common knowledge of this fact. This would creates an unprecedented level of trust. (The only possible exception I can think of are twins. And even so…)
So, let’s recap:
Thinks 100 times faster than a human, though no better.
Can copy itself over many times (the exact amount depends on computing power available).
The resulting team forms a nearly perfectly coordinated group.
Do you at least concede that this is potentially more dangerous than a whole country armed up with nukes? Would you rely on it being less dangerous than that?
When I imagine that I could make my copy which would be identical to me, sharing my goals, able to copy its experiences back to me, and willing to die for me (something like Naruto’s clones), taking over the society seems rather easy. (Assuming that no one else has this ability, and no one suspects me of having it. In real life it would probably help if all the clones looked different, but had an ability to recognize each other.)
Research: For each interesting topic I could make dozen clones which would study the topic in libraries and universities, and discuss their findings with each other. I don’t suppose it would make me an expert on everything, but I could get at least all the university-level education on most things.
Resources: If I can make more money than I spend, and if I don’t make too much copies to imbalance the economy, I can let a few dozen clones work and produce the money for the rest of them. At least in the starting phase, until my research groups discover better ways to make money.
Contacts: Different clones could go to different places making useful contacts wil different kinds of people. Sometimes you find a person which can help your goals significantly. With many clones I could make contacts in many different social groups, and overcome language or religious barriers (I can have a few clones learn the language or join the religion).
Multiple “first impressions”: If I need a help of a given person or organization, I could in many cases gain their trust by sending multiple different clones to them, using different strategies to befriend them, until I find one that works.
Taking over democratic organizations: Any organization with low barriers to entry and democratic voting can be taken over by sending enough clones there, and then voting some of the clones as new leaders. A typical non-governmental organization or even a smaller political party could be gained this way. I don’t even need a majority of clones there: two potential leaders competing with each other, half dozen experts openly supporting each of them, and dozen people befriending random voters and explaining them why leader X or leader Y is the perfect choice; then most of the voting would be done by other people.
Assassination: If someone is too much of a problem, I can create a clone which kills them and then disappears. This should be used very rarely, not to draw attention to my abilities.
Safety: To protect myself, I would send my different clones to different countries over the world.
Joining all the winning sides: If there is an important group of people, I could join them, even the groups fighting against each other. Whoever wins, some of my clones are on the winning side.
There are a lot of “ifs”, though.
If that AI runs on expensive or specialized hardware, it can’t necessarily expand much. For instance, if it runs on hardware worth millions of dollars, it can’t exactly copy itself just anywhere yet. Assuming that the first AI of that level will be cutting edge research and won’t be cheap, that gives a certain time window to study it safely.
The AI may be dangerous if it appeared now, but if it appears in, say, fifty years, then it will have to deal with the state of the art fifty years from now. Expanding without getting caught might be considerably more difficult then than it is now—weak AI will be all over the place, for one.
Last, but not least, the AI must have access to its own source code in order to copy it. That’s far from a given, especially if it’s a neural architecture. A human-level AI would not know how it works any more than we know how we work, so if it has no read access to itself or no way to probe its own circuitry, it won’t be able to copy itself at all. I doubt the first AI would actually have fine-grained access to its own inner workings, and I doubt it would have anywhere close to the amount of resources required to reverse engineer itself. Of course, that point is moot if some fool does give it access...
I agree with your first point, though it gets worse for us as hardware gets cheaper and cheaper.
I like your second point even more: it’s actionable. We could work on the security of personal computers.
That last one is incorrect however. The AI only have to access its object code in order to copy itself. That’s something even current computer viruses can do. And we’re back to boxing it.
If the AI is a learning system such as a neural network, and I believe that’s quite likely to be the case, there is no source/object dichotomy at all and the code may very well be unreadable outside of simple local update procedures that are completely out of the AI’s control. In other words, it might be physically impossible for both the AI and ourselves to access the AI’s object code—it would be locked in a hardware box with no physical wires to probe its contents, basically.
I mean, think of a physical hardware circuit implementing a kind of neuron network—in order for the network to be “copiable”, you need to be able to read the values of all neurons. However, that requires a global clock (to ensure synchronization, though AI might tolerate being a bit out of phase) and a large number of extra wires connecting each component to busses going out of the system. Of course, all that extra fluff inflates the cost of the system, makes it bigger, slower and probably less energy efficient. Since the first human-level AI won’t just come out of nowhere, it will probably use off-the-shelf digital neural components, and for cost and speed reasons, these components might not actually offer any way to copy their contents.
This being said, even if the AI runs on conventional hardware, locking it out of its own object code isn’t exactly rocket science. The specification of some programming languages already guarantee that this cannot happen, and type/proof theory is an active research field that may very well be able to prove the conformance of implementation to specification. If the AI is a neural network emulated on conventional hardware, the risks that it can read itself without permission are basically zilch.
What we usually mean by intelligence doesn’t include the skills necessary for getting to be in charge.
There are various notions of intelligence, social intelligence includes the skills for getting in charge.
My point is that human-level intelligence, even replicated or sped up, is generally not enough.
I’m not sure if that’s a good comparison. Compare the following cases:
A. 1 smart human, given 100 days to solve some problem
B. 100 smart humans, given 1 day to solve some problem.
C. 1,000 smart humans, given 1 day to solve some problem.
A would outperform B on most tasks, and probably even C. Most problems just aren’t that parallelizable.
That’s why I wrote “or a little more, but within the same order or magnitude”