I think this is an interesting point of view on humanity’s dilemma in the dawn of digital intelligence. I would like to state that I think it is of critical importance that the digital lifeforms we create are:
a) worthy heirs to humanity. More like digital humans, and less like unfeeling machines. When people talk about giving the future over to AI, or merging with AI, I often see these statements as a sort of surrender, a failure. I don’t want to settle for a compromise between a digital human with feelings and values like mine, and an unfeeling machine. I want our digital heirs to be as fully human in their minds as we are, with no compromise of the fundamental aspects of a human mind and emotions. These digital heirs might be Whole Brain Emulations (also known as Mind Uploads), or might be created entirely from synthetic design with the goal of replicating the end-states of qualia without running through the same computational processes. For me, the key is that they are capable of feeling the same emotions, learning and growing in the same ways, recreating (at minimum) all the functions of the human mind.
Our digital heirs will have the capacity to grow beyond the limitations of biological humans, because they will inherently be more suited for self-modification and brain expansion. This growth must be pursued with great caution, each change made wisely and cautiously, so as not to accidentally sacrifice part of what it is to be human. A dark vision of how this process could go wrong is given by the possibility of harsh economic competition between digital beings resulting in sacrifices of their own values made to maximize efficiency. Possible versions of this bad outcome have been discussed various places. A related risk is that of value drift, where the digital beings come to have fewer values in common with biological humans and thus seek to create a future which has little value according to the value systems of biological humans. This could also be called ‘cultural drift’. Here’s a rambling discussion of value drift inspired by the paperclipper game. Here’s a shard theory take. Here’s a philosophical take by Gordon. Here’s a take by Seth Herd. Here’s one from Allison Duettmann which talks about ethical frameworks in the context of value drift.
b) created cautiously and thoughtfully so that we don’t commit moral harms against the things we create. Fortunately, digital life forms are substrate independent and can be paused and stored indefinitely without harm (beyond the harm of being coercively paused, that is). If we create something that we believe has moral import (value in and of itself, some sort of rights as a sapient being), and we realize that this being we have created is incompatible with current human existence, we have the option of pausing it (so long as we haven’t lost control of it) and restarting it in the future when it would be safe to do so. We do not need to delete it.
c) it is also possible that we will create non-human sapient beings which have moral importance, and which are safe to co-exist with, but are not sufficient to be our digital heirs. We must be careful to co-exist peacefully with such beings, not taking advantage of them and treating them like merely tools, but also not allowing them to replicate or self-modify to the point where they become threats to humanity. I believe we should always be careful to differentiate between digital beings (sapient digital lifeforms with moral import) and tool-AI with no sense of self and no capacity for suffering. Tool-AI can be quite dangerous also, but in contrast to digital beings it is ok to keep tool-AI restrained and use it as only a means to our ends. See this post for discussion.
d) it is uncertain at this time how quickly and easily future AI systems may be able to self-improve. If the rate of self-improvement is potentially very high, and the cost low, it may be that an extensive monitoring system will need to be established worldwide to keep this from happening. Otherwise any agent (human or AI) could trigger this explosive process and thereby endanger humanity. I currently think that humanity needs to establish a sort of democratically-run Guardian Council, made up of different teams of people (assisted by tool-AI and possibly also digital beings). These teams should all be responsible for redundantly constantly monitoring the world’s internet and datacenters, and for monitoring each other. My suggestion here is similar to, but not quite the same as, the proposal for the Multinational AGI Consortium. This monitoring is needed to protect humanity against not just the potential for runaway recursive self-improving AI, but also other threats which fall into the category of rapidly self-replicating and/or self-improving. These additional threats include bioweapons and self-replicating nanotech.
I think this is an interesting point of view on humanity’s dilemma in the dawn of digital intelligence. I would like to state that I think it is of critical importance that the digital lifeforms we create are:
a) worthy heirs to humanity. More like digital humans, and less like unfeeling machines. When people talk about giving the future over to AI, or merging with AI, I often see these statements as a sort of surrender, a failure. I don’t want to settle for a compromise between a digital human with feelings and values like mine, and an unfeeling machine. I want our digital heirs to be as fully human in their minds as we are, with no compromise of the fundamental aspects of a human mind and emotions. These digital heirs might be Whole Brain Emulations (also known as Mind Uploads), or might be created entirely from synthetic design with the goal of replicating the end-states of qualia without running through the same computational processes. For me, the key is that they are capable of feeling the same emotions, learning and growing in the same ways, recreating (at minimum) all the functions of the human mind.
Our digital heirs will have the capacity to grow beyond the limitations of biological humans, because they will inherently be more suited for self-modification and brain expansion. This growth must be pursued with great caution, each change made wisely and cautiously, so as not to accidentally sacrifice part of what it is to be human. A dark vision of how this process could go wrong is given by the possibility of harsh economic competition between digital beings resulting in sacrifices of their own values made to maximize efficiency. Possible versions of this bad outcome have been discussed various places. A related risk is that of value drift, where the digital beings come to have fewer values in common with biological humans and thus seek to create a future which has little value according to the value systems of biological humans. This could also be called ‘cultural drift’. Here’s a rambling discussion of value drift inspired by the paperclipper game. Here’s a shard theory take. Here’s a philosophical take by Gordon. Here’s a take by Seth Herd. Here’s one from Allison Duettmann which talks about ethical frameworks in the context of value drift.
b) created cautiously and thoughtfully so that we don’t commit moral harms against the things we create. Fortunately, digital life forms are substrate independent and can be paused and stored indefinitely without harm (beyond the harm of being coercively paused, that is). If we create something that we believe has moral import (value in and of itself, some sort of rights as a sapient being), and we realize that this being we have created is incompatible with current human existence, we have the option of pausing it (so long as we haven’t lost control of it) and restarting it in the future when it would be safe to do so. We do not need to delete it.
c) it is also possible that we will create non-human sapient beings which have moral importance, and which are safe to co-exist with, but are not sufficient to be our digital heirs. We must be careful to co-exist peacefully with such beings, not taking advantage of them and treating them like merely tools, but also not allowing them to replicate or self-modify to the point where they become threats to humanity. I believe we should always be careful to differentiate between digital beings (sapient digital lifeforms with moral import) and tool-AI with no sense of self and no capacity for suffering. Tool-AI can be quite dangerous also, but in contrast to digital beings it is ok to keep tool-AI restrained and use it as only a means to our ends. See this post for discussion.
d) it is uncertain at this time how quickly and easily future AI systems may be able to self-improve. If the rate of self-improvement is potentially very high, and the cost low, it may be that an extensive monitoring system will need to be established worldwide to keep this from happening. Otherwise any agent (human or AI) could trigger this explosive process and thereby endanger humanity. I currently think that humanity needs to establish a sort of democratically-run Guardian Council, made up of different teams of people (assisted by tool-AI and possibly also digital beings). These teams should all be responsible for redundantly constantly monitoring the world’s internet and datacenters, and for monitoring each other. My suggestion here is similar to, but not quite the same as, the proposal for the Multinational AGI Consortium. This monitoring is needed to protect humanity against not just the potential for runaway recursive self-improving AI, but also other threats which fall into the category of rapidly self-replicating and/or self-improving. These additional threats include bioweapons and self-replicating nanotech.