What evolutionary pressures? Even making the incredible assumption that we will continue to use sequences of genes as a large part of our identities, what’s to stop a singleton of some variety from eliminating drives for replication or expansion entirely?
I feel uncomfortable speculating about a post-machine-intelligence future even to this extent; this is not a realm in which I am confident about any proposition. Consequently, I view all confident conclusions with great skepticism.
You’re still not getting the breadth and generality of Hanson’s model. To use recent LW terminology, it’s an anti-prediction.
It doesn’t matter whether agents perpetuate their strategies by DNA mixing, binary fission, cellular automata, or cave paintings. Even if all but a tiny minority of posthumans self-modify not to want growth or replication, the few that don’t will soon dominate the light-cone. A singleton, like I’d mentioned, is one way to avert this. Universal extinction and harsh, immediate punishment of expansion-oriented agents are the only others I see.
You (or Robin, I suppose) are just describing a many-agent prisoner’s dilemma. If TDT agents beat the dilemma by cooperating with other TDT agents, then any agents that started out with a different decision theory will have long since self-modified to use TDT.
Alternately, if there is no best decision theoretical solution to the prisoner’s dilemma, then we probably don’t need to worry about surviving to face this problem.
Now, there’s a generalized answer. It even covers the possibility of meeting aliens—finding TDT is a necessary condition for reaching the stars. Harsh punishment of inconsiderate expanders might still be required, but there could be a stable equilibrium without ever actually inflicting that punishment. That’s a new perspective for me, thanks!
To be even more general, suppose that there is at least one thing X that is universally necessary for effective superintelligences to function. X might be knowledge of the second law of thermodynamics, TDT, a computational substrate of some variety, or any number of other things. There are probably very many such X’s, many of which are entirely non-obvious to any entity that is not itself a superintelligence (i.e. us). Furthermore, there may be at least one thing Y that is universally incompatible with effective superintelligence. Y might be an absolute belief in the existence of the deity Thor or desiring only to solve the Halting Problem using a TM-equivalent. For the Hansonian model to hold, all X’s and no Y’s must be compatible with the desire and ability to expand and/or replicate.
This argument is generally why I dislike speculating about superintelligences. It is impossible for ordinary humans to have exhaustive (or even useful, partial) knowledge of all X and all Y. The set of all things Y in particular may not even be enumerable.
We cannot be sure that there are difficulties beyond our comprehension but we are certainly able to assign probabilities to that hypothesis based on what we know. I would be justifiably shocked if something we could call a super-intelligence couldn’t be formed based on knowledge that is accessible to us, even if the process of putting the seed of a super-intelligence together is beyond us.
Humans aren’t even remotely optimised for generalised intelligence, it’s just a trick we picked up to, crudely speaking, get laid. There is no reason that a intelligence of the form “human thinking minus the parts that suck and a bit more of the parts that don’t suck” couldn’t be created using the knowledge available to us and that is something we can easily place a high probability on. Then you run the hardware at more than 60hz.
Oh, I agree. We just don’t know what self-modifications will be necessary to achieve non-speed-based optimizations.
To put it another way, if superintelligences are competing with each other and self-modifying in order to do so, predictions about the qualities those superintelligences will possess are all but worthless.
To put it another way, if superintelligences are competing with each other and self-modifying in order to do so, predictions about the qualities those superintelligences will possess are all but worthless.
What evolutionary pressures? Even making the incredible assumption that we will continue to use sequences of genes as a large part of our identities, what’s to stop a singleton of some variety from eliminating drives for replication or expansion entirely?
Your point is spot on. Competition can not be relied on to produce adaptation if someone wins the competition once and for all.
What evolutionary pressures? Even making the incredible assumption that we will continue to use sequences of genes as a large part of our identities, what’s to stop a singleton of some variety from eliminating drives for replication or expansion entirely?
I feel uncomfortable speculating about a post-machine-intelligence future even to this extent; this is not a realm in which I am confident about any proposition. Consequently, I view all confident conclusions with great skepticism.
You’re still not getting the breadth and generality of Hanson’s model. To use recent LW terminology, it’s an anti-prediction.
It doesn’t matter whether agents perpetuate their strategies by DNA mixing, binary fission, cellular automata, or cave paintings. Even if all but a tiny minority of posthumans self-modify not to want growth or replication, the few that don’t will soon dominate the light-cone. A singleton, like I’d mentioned, is one way to avert this. Universal extinction and harsh, immediate punishment of expansion-oriented agents are the only others I see.
You (or Robin, I suppose) are just describing a many-agent prisoner’s dilemma. If TDT agents beat the dilemma by cooperating with other TDT agents, then any agents that started out with a different decision theory will have long since self-modified to use TDT.
Alternately, if there is no best decision theoretical solution to the prisoner’s dilemma, then we probably don’t need to worry about surviving to face this problem.
Now, there’s a generalized answer. It even covers the possibility of meeting aliens—finding TDT is a necessary condition for reaching the stars. Harsh punishment of inconsiderate expanders might still be required, but there could be a stable equilibrium without ever actually inflicting that punishment. That’s a new perspective for me, thanks!
To be even more general, suppose that there is at least one thing X that is universally necessary for effective superintelligences to function. X might be knowledge of the second law of thermodynamics, TDT, a computational substrate of some variety, or any number of other things. There are probably very many such X’s, many of which are entirely non-obvious to any entity that is not itself a superintelligence (i.e. us). Furthermore, there may be at least one thing Y that is universally incompatible with effective superintelligence. Y might be an absolute belief in the existence of the deity Thor or desiring only to solve the Halting Problem using a TM-equivalent. For the Hansonian model to hold, all X’s and no Y’s must be compatible with the desire and ability to expand and/or replicate.
This argument is generally why I dislike speculating about superintelligences. It is impossible for ordinary humans to have exhaustive (or even useful, partial) knowledge of all X and all Y. The set of all things Y in particular may not even be enumerable.
We cannot be sure that there are difficulties beyond our comprehension but we are certainly able to assign probabilities to that hypothesis based on what we know. I would be justifiably shocked if something we could call a super-intelligence couldn’t be formed based on knowledge that is accessible to us, even if the process of putting the seed of a super-intelligence together is beyond us.
Humans aren’t even remotely optimised for generalised intelligence, it’s just a trick we picked up to, crudely speaking, get laid. There is no reason that a intelligence of the form “human thinking minus the parts that suck and a bit more of the parts that don’t suck” couldn’t be created using the knowledge available to us and that is something we can easily place a high probability on. Then you run the hardware at more than 60hz.
Oh, I agree. We just don’t know what self-modifications will be necessary to achieve non-speed-based optimizations.
To put it another way, if superintelligences are competing with each other and self-modifying in order to do so, predictions about the qualities those superintelligences will possess are all but worthless.
On this I totally agree!
Your point is spot on. Competition can not be relied on to produce adaptation if someone wins the competition once and for all.