The natural consequence of “postbiological humans” is effective disempowerment if not extinction of humanity as a whole.
Such “transhumanists” clearly do not find the eradication of biology abhorrent, any more than any normal person would find the idea of “substrate independence”(death of all love and life) to be abhorrent.
Value is based on scarcity. That which can be copied and pasted has little value.
In any story, this is the equivalent of discussing why undeath would be better than life.
All of this seems to be a higher value world to me than either a world of “artificial people” which thus ends the entire cycle of life itself or total extinction of humanity, which is likely also as a result of AI continuity.
As such, it seems that total human consciousness may endure longer, tell and feel more stories and thus have a higher total existence by having a near total catastrope to lower the rate of AI development.
Disagree: values come from substrate and environmental. I would almost certainly ally myself with biological aliens versus a digital “humanity” as the biological factor will create a world of much more reasonable values to me.
We do have world takeover compared to ants, though our desire to wipe out all ants is just not that high.
I think even if AI proves strictly incapable of surviving in the long time due to various efficiency constraints, this has no relevance on its ability to kill us all.
A paperclip maximizer that eventually runs into a halting problem as it tries to paperclip itself may very well have killed everyone by that point.
I think the term for this is “minimal viable exterminator.”
But land and food doesnt actually give you more computational capability: only having another human being cooperate with you in some way can.
The essential point here is that values depend upon the environment and the limitations thereof, so as you change the limitations, the values change. The values important for a deep sea creature with extremely limited energy budget, for example, will be necessarily different from that of human beings.
Humans can’t eat another human and get access to the victim’s data and computation but AI can. Human cooperation is a value created by our limitations as humans, which AI does not have similar constraints for.
I disagree on the inference to the recent post, which I quite liked and object heavily to Hanson’s conclusions.
The ideal end state is very different: in the post mentioned, biological humans, if cyborgs, are in control. The Hanson endpoint has only digital emulations of humanity.
This is the basic distinguishing point between the philosophies of Cyborgism vs more extreme ones like mind uploading or Hanson’s extinction of humanity as we know it for “artificial descendants.”
Both open thread links at the base of the article lead to errors for me.
“You can’t reason a man out of a position he has never reasoned himself into.”
I think I have seen a similar argument on LW for this, and it is sensible. With vast intelligence, it is possible for the search space to support priors to be even greater. An AI with a silly but definite value like “the moon is great, I love the moon” may not change its value as much as develop an entire religion around the greatness of the moon.
We see this in goal misgeneralization, where it very much maximizes a reward function independent of the meaningful goal.
I have considered the loss of humanity from being in a hive mind versus the loss of humanity from being extinct completely or being emulated on digital processes, and concluded as bad as it might be to become much more akin to true eusocial insects like ants, you still have more humanity left by keeping some biology and individual bodies.
But if you believed that setting fire to everything around you was good, and by showing you that hurting ecosystems by fire would be bad, you would change your values, would that really be “changing your values?”
A lot of values update based on information, so perhaps one could realign such AI with such information.
I have never had much patience for Hanson and it seems someone as intelligent as himself should know that values emerge from circumstance. What use, for example, would AI have for romantic love in a world where procreation consists of digital copies? What use are coordinated behaviors for society if lies are impossible and you can just populate your “society” with clones of yourself? What use is there for taste without the evolutionary setup for sugars, etc.
Behaviors arise from environmental conditions, and its just wild to see a thought that eliminating all of that would give us anything similar.
Essentially the only value you will preserve is the universal one for power seeking. I like to think very few of us want to value power seeking over love and cooperation: right now, Hanson is valuing the “power” of his “descendants” over their ability to be human, why would AI be different?
I also believe that animal life and cognition has value, as their own form of non-human intelligence. An AI catastrope that eliminates the biosphere seems to be vastly negative, immoral and agency-reducing for them: they didn’t vote to go extinct.
I count myself among the simple and the issue would seem to be that I would just take the easiest solution of not building a doom machine, to minimize risks of temptation.
Or as the Hobbits did, throw the Ring into a volcano, saving the world the temptation. Currently, though, I have no way of pressing a button to stop it.
I believe that the general consensus is that it is impossible to totally pause AI development due to Molochian concerns: I am like you, and if I could press a button to send us back to 2017 levels of AI technology, I would.
However, in the current situation, the intelligent people as you noted have found ways to convince themselves to take on a very high risk of humanity and the general coordination of humanity is not enough to convince them otherwise.
There have some positive updates but it seems that we have not been in a world of general sanity and safety at this scale.
I have taken solace in the morbid amusement that many of the “powers that be” may indeed be dying with us, but they are quite blind.
“Humanity: extinct due to hyperbolic discounting behavior.”
So in short, they are generally unconcerned with existential risks? I’ve spoken with some staff and I get the sense they do not believe it will impact them personally.
I would prefer total oblivion over AI replacement myself: complete the Fermi Paradox.
I have been wondering if the new research into organoids will help? It would seem one of the easiest ways to BCI is to use more brain cells.
One example would be the below:
This solves nothing that could not be better solved by freezing development of hardware, which would also slow down evolutionary setups.
This also allows for more time for safer approaches such as genetic engineering and biological advancements to catch up, and keep us from Killing Everyone.