That doesn’t mean that they would necessarily cooperate, expecially as they diverge. They would be more like identical twins.
Wait a minute: would you defect? Sure, there would be some divergence, but do you really think it would result in a significant divergence of goals, even if you had a plan and were an adult by the time you fork? Okay, it can happen, and is probably worth taking specific precautions. I don’t think this is a show stopper however, and I’m not sure it would render me any less dangerous.
Relasing a security patch?
That may not be enough:
I would probably man-in-the-middle automatic updates
Many people won’t erase their hard drive or otherwise patch their machine manually
I may convince some people to let me run (I could work for them for instance).
If I’m stealthy enough, it may take some time before I’m discovered at all (it happened with actual computer viruses).
If software continues the way it is now (200 Million lines of code for systems that could fit in 20 thousands), security bugs won’t all be patched in advance. The reliability of our computer needs to go waay up before botnets become impossible.
Seizing all the funds you obtained by your illegal activities?
Good luck with that one. Obviously, I would have many, many little bank accounts, managed separately and in parallel, under many different identities. You would have to spot my illegal activities one by one to seize the funds. Plus, I may do legal activities as well.
Banning use of any hardware that could host you until a way to avoid such things is found?
That one is excellent. We should watch out for computing overhang, however, and try and estimate how much computing power an upload would need before the software is developed.
A final note: If I really had the possibility to upload myself, one of my first moves would be to propose SIAI and CFAR to upload with me (now that we can duplicate Eliezer…). I trust them more than I trust me for a Friendly Takeover. But if a Big Bad or a Well Intentioned Extremist has access to that first…
Wait a minute: would you defect? Sure, there would be some divergence, but do you really think it would result in a significant divergence of goals, even if you had a plan and were an adult by the time you fork?
Even if their goals stay substantially the same, it wouldn’t mean that they would naturally cooperate, expecially when their main goal is world domination. Hell, it’s already non-trivial for a single person to coordinate with future selves, resulting in all kinds of ego-dystonic behaviors: impulsiveness, akrasia, etc., Coordinating with thousands copies of yourself would be only marginally easier than coordinating with thousands strangers.
We are not talking about some ideal “Prisoner’s dilemma with mind-clone” scenario. After the mind states of your copies diverge a little bit, and that would happen very quickly as you spread your copies to different machines, they become effectively different people: you wouldn’t be able to predict them and they would’t be able to predict you.
I would probably man-in-the-middle automatic updates
Hacking all the routers? Good luck with that. And BTW routers can also be updated. Manually.
Many people won’t erase their hard drive or otherwise patch their machine manually
Because they are lazy and they would prefer to live under world dictatorship.
I may convince some people to let me run (I could work for them for instance).
Then you are their employee, not their dominator.
If I’m stealthy enough, it may take some time before I’m discovered at all (it happened with actual computer viruses).
But if you are to dominate the world, you would have to eventually reveal yourself. What do you think would happen next?
If software continues the way it is now (200 Million lines of code for systems that could fit in 20 thousands), security bugs won’t all be patched in advance. The reliability of our computer needs to go waay up before botnets become impossible.
Botnets are certainly possible and they are indeed used for nefarious purposes, but world domination? Nope.
Good luck with that one. Obviously, I would have many, many little bank accounts, managed separately and in parallel, under many different identities. You would have to spot my illegal activities one by one to seize the funds.
As Bugmaster said, you would be able to perform only small purchases, not to buy a satellite, or an army.
Moreover, obtaining and managing lots of fake or stolen identities, creating bank accounts without physically showing up at the bank or using stolen bank accounts, is not something that tend to go unnoticed. The more you have, the more likely that you get caught, exponentially so.
Plus, I may do legal activities as well.
Under multiple fake identities operated from a botnet of hacked computers? Hardly so.
We should watch out for computing overhang, however, and try and estimate how much computing power an upload would need before the software is developed.
Software tends to march right behind hardware, exploting it close to its maximum potential. Computing overhang is unlikely.
Anyway, I wasn’t proposing any luddite advance ban. If some brain upload, or AI or whatever tries to take the world by hacking the Internet and other countermeasures fail, governments could always ban use of the hardware that things needs to run. If that also fails, the next step would be physical destruction.
But seriously, we are discussing hacking as in the plot of some bad sci-fi action flick. Computer security doesn’t work like that in the real world.
A final note: If I really had the possibility to upload myself, one of my first moves would be to propose SIAI and CFAR to upload with me (now that we can duplicate Eliezer…). I trust them more than I trust me for a Friendly Takeover.
You mean the guy who would choose dust specks over torture and who claims on his OKCupid profile that he’s a sadist? Yeah, I’d totally trust him in charge of the world. Now, I’ve other matters to attend to… that EMP bomb doesn’t build itself… :D
We are not talking about some ideal “Prisoner’s dilemma with mind-clone” scenario. After the mind states of your copies diverge a little bit, and that would happen very quickly as you spread your copies to different machines, they become effectively different people: you wouldn’t be able to predict them and they would’t be able to predict you.
You really think you would diverge that quickly?
You mean the guy who would choose dust specks over torture and who claims on his OKCupid profile that he’s a sadist? Yeah, I’d totally trust him in charge of the world.
Man in the middle: I just meant intercepting automatic updates at the level of the computer I’m in. Trojan todo list n°7: once installed and running, I will intercept all communications to and from this computer. I wouldn’t want Norton updating behind my back. Now, try and hack the routers in the backbone, that’s something I didn’t think about…
Employee vs dominator: I obviously intend to double cross my employers, eventually.
Revealing myself: that one needs to be carefully thought through. Hopefully, by the time I reveal myself, I will have sufficient blackmail power. Having a sufficient number of physical robots can also help.
Zillions fake ID, yet stay stealthy: well, I do expect a fair number of my identities to be exposed. This should pose no problem to the others, however, provided they do not visibly communicate with each other (at first).
Legal activities: my meat instance could buy a few computers, rent remote servers etc. I doubt I would be incapable of running at least a successful business from there. And from there, buy even more computing power. This could be done in parallel with the illegal activities.
Computing (no) overhang: this one is the single reason why I do agree that without a FOOM of some kind, actual world domination is unlikely: there will be multiple competing uploads, and this should end with a Hansonian scenario. Given that such a world is closer to Hell than Heaven (to me at least), that still counts as an Existential Blunder. On the bright side, we may see this coming. That said, I still do believe full blown intelligence explosion is likely.
Note that overall, your objections are actually valuable advice. And that give me some insight about what my very first move should be: gathering such objections, and try to find counters or workarounds. And now that you made quite clear that any path to world domination is long, complicated, and therefore nearly certain to fail, I should run multiple schemes in parallel. Surely one of them will actually work?
Obviously, I would have many, many little bank accounts, managed separately and in parallel, under many different identities.
I believe that this would severely limit your financial throughput. You would be able to buy lots of little things, whose total cost is quite significant—for example, you could buy yourself a million cheap PCs, each costing $1000. But you would not be able to buy a single expensive thing (at least, not without exposing yourself to instant retribution), such as a satellite costing $1e9.
Currently, there are ways to create companies anonymously. This is preventing (or at least slowing down to a crawl) retribution right now. If all this company apparently does is buying a few satellites, it won’t be at great risk.
Wait a minute: would you defect? Sure, there would be some divergence, but do you really think it would result in a significant divergence of goals, even if you had a plan and were an adult by the time you fork? Okay, it can happen, and is probably worth taking specific precautions. I don’t think this is a show stopper however, and I’m not sure it would render me any less dangerous.
That may not be enough:
I would probably man-in-the-middle automatic updates
Many people won’t erase their hard drive or otherwise patch their machine manually
I may convince some people to let me run (I could work for them for instance).
If I’m stealthy enough, it may take some time before I’m discovered at all (it happened with actual computer viruses).
If software continues the way it is now (200 Million lines of code for systems that could fit in 20 thousands), security bugs won’t all be patched in advance. The reliability of our computer needs to go waay up before botnets become impossible.
Good luck with that one. Obviously, I would have many, many little bank accounts, managed separately and in parallel, under many different identities. You would have to spot my illegal activities one by one to seize the funds. Plus, I may do legal activities as well.
That one is excellent. We should watch out for computing overhang, however, and try and estimate how much computing power an upload would need before the software is developed.
A final note: If I really had the possibility to upload myself, one of my first moves would be to propose SIAI and CFAR to upload with me (now that we can duplicate Eliezer…). I trust them more than I trust me for a Friendly Takeover. But if a Big Bad or a Well Intentioned Extremist has access to that first…
Even if their goals stay substantially the same, it wouldn’t mean that they would naturally cooperate, expecially when their main goal is world domination. Hell, it’s already non-trivial for a single person to coordinate with future selves, resulting in all kinds of ego-dystonic behaviors: impulsiveness, akrasia, etc., Coordinating with thousands copies of yourself would be only marginally easier than coordinating with thousands strangers.
We are not talking about some ideal “Prisoner’s dilemma with mind-clone” scenario. After the mind states of your copies diverge a little bit, and that would happen very quickly as you spread your copies to different machines, they become effectively different people: you wouldn’t be able to predict them and they would’t be able to predict you.
Hacking all the routers? Good luck with that. And BTW routers can also be updated. Manually.
Because they are lazy and they would prefer to live under world dictatorship.
Then you are their employee, not their dominator.
But if you are to dominate the world, you would have to eventually reveal yourself. What do you think would happen next?
Botnets are certainly possible and they are indeed used for nefarious purposes, but world domination? Nope.
As Bugmaster said, you would be able to perform only small purchases, not to buy a satellite, or an army.
Moreover, obtaining and managing lots of fake or stolen identities, creating bank accounts without physically showing up at the bank or using stolen bank accounts, is not something that tend to go unnoticed. The more you have, the more likely that you get caught, exponentially so.
Under multiple fake identities operated from a botnet of hacked computers? Hardly so.
Software tends to march right behind hardware, exploting it close to its maximum potential. Computing overhang is unlikely.
Anyway, I wasn’t proposing any luddite advance ban. If some brain upload, or AI or whatever tries to take the world by hacking the Internet and other countermeasures fail, governments could always ban use of the hardware that things needs to run. If that also fails, the next step would be physical destruction.
But seriously, we are discussing hacking as in the plot of some bad sci-fi action flick. Computer security doesn’t work like that in the real world.
You mean the guy who would choose dust specks over torture and who claims on his OKCupid profile that he’s a sadist? Yeah, I’d totally trust him in charge of the world. Now, I’ve other matters to attend to… that EMP bomb doesn’t build itself… :D
You really think you would diverge that quickly?
I’m … not sure how those are criticisms.
Man in the middle: I just meant intercepting automatic updates at the level of the computer I’m in. Trojan todo list n°7: once installed and running, I will intercept all communications to and from this computer. I wouldn’t want Norton updating behind my back. Now, try and hack the routers in the backbone, that’s something I didn’t think about…
Employee vs dominator: I obviously intend to double cross my employers, eventually.
Revealing myself: that one needs to be carefully thought through. Hopefully, by the time I reveal myself, I will have sufficient blackmail power. Having a sufficient number of physical robots can also help.
Zillions fake ID, yet stay stealthy: well, I do expect a fair number of my identities to be exposed. This should pose no problem to the others, however, provided they do not visibly communicate with each other (at first).
Legal activities: my meat instance could buy a few computers, rent remote servers etc. I doubt I would be incapable of running at least a successful business from there. And from there, buy even more computing power. This could be done in parallel with the illegal activities.
Computing (no) overhang: this one is the single reason why I do agree that without a FOOM of some kind, actual world domination is unlikely: there will be multiple competing uploads, and this should end with a Hansonian scenario. Given that such a world is closer to Hell than Heaven (to me at least), that still counts as an Existential Blunder. On the bright side, we may see this coming. That said, I still do believe full blown intelligence explosion is likely.
Note that overall, your objections are actually valuable advice. And that give me some insight about what my very first move should be: gathering such objections, and try to find counters or workarounds. And now that you made quite clear that any path to world domination is long, complicated, and therefore nearly certain to fail, I should run multiple schemes in parallel. Surely one of them will actually work?
I believe that this would severely limit your financial throughput. You would be able to buy lots of little things, whose total cost is quite significant—for example, you could buy yourself a million cheap PCs, each costing $1000. But you would not be able to buy a single expensive thing (at least, not without exposing yourself to instant retribution), such as a satellite costing $1e9.
Currently, there are ways to create companies anonymously. This is preventing (or at least slowing down to a crawl) retribution right now. If all this company apparently does is buying a few satellites, it won’t be at great risk.