It is good to have deontological commitments about what you would do with a lot of power. But this situation is very different from “a lot of power”, it’s also “if you were to become wiser and more knowledgeable than anyone in history so far”. One can imagine the Christians of old asking for a commitment that “If you get this new scientific and industrial civilization that you want in 2,000 years from now, will you commit to following the teachings of Jesus?” and along the way I sadly find out that even though it seemed like a good and moral commitment at the time, it totally screwed my ability to behave morally in the future because Christianity is necessarily predicated on tons of falsehoods and many of its teachings are immoral.
But there is some version of this commitment I think might be good to make… something like “Insofar as the players involved are all biological humans, I will respect the legal structures that exist and the existence of countries, and will not relate to them in ways that would be considered worthy of starting a war in its defense”. But I’m not certain about this, for instance what if most countries in the world build 10^10 digital minds and are essentially torturing them? I may well wish to overthrow a country that is primarily torture with a small number of biological humans sitting on thrones on top of these people, and I am not willing to commit not to do that presently.
I understand that there are bad ethical things one can do with post-singularity power, but I do not currently see a clear way to commit to certain ethical behaviors that will survive contact with massive increases in knowledge and wisdom. I am interested if anyone has made other commitments about post-singularity life (or “on the cusp of singularity life”) that they expect to survive contact with reality?
Added: At the very least I can say that I am not going to make commitments to do specific things that violate my current ethics. I have certainly made no positive commitment to violate people’s bodily autonomy nor have such an intention.
Fair, I also haven’t made any specific commitments, I phrased it wrongly. I agree there can be extreme scenarios with trillions of digital minds tortured where you’d maybe want to declare war on the. rest of society. But I would still like people to write down that “of course, I wouldn’t want to destroy Earth before we can save all the people who want to live in their biological bodies, just to get a few years of acceleration in the cosmic conquest”. I feel a sentence like this should really have been included in the original post about dismantling the Sun, and until people are not willing to write this down, I remain paranoid that they would in fact haul the Amish the extermination camps if it feels like a good idea at the time. (As I said, I met people who really held this position.)
It is good to have deontological commitments about what you would do with a lot of power. But this situation is very different from “a lot of power”, it’s also “if you were to become wiser and more knowledgeable than anyone in history so far”. One can imagine the Christians of old asking for a commitment that “If you get this new scientific and industrial civilization that you want in 2,000 years from now, will you commit to following the teachings of Jesus?” and along the way I sadly find out that even though it seemed like a good and moral commitment at the time, it totally screwed my ability to behave morally in the future because Christianity is necessarily predicated on tons of falsehoods and many of its teachings are immoral.
But there is some version of this commitment I think might be good to make… something like “Insofar as the players involved are all biological humans, I will respect the legal structures that exist and the existence of countries, and will not relate to them in ways that would be considered worthy of starting a war in its defense”. But I’m not certain about this, for instance what if most countries in the world build 10^10 digital minds and are essentially torturing them? I may well wish to overthrow a country that is primarily torture with a small number of biological humans sitting on thrones on top of these people, and I am not willing to commit not to do that presently.
I understand that there are bad ethical things one can do with post-singularity power, but I do not currently see a clear way to commit to certain ethical behaviors that will survive contact with massive increases in knowledge and wisdom. I am interested if anyone has made other commitments about post-singularity life (or “on the cusp of singularity life”) that they expect to survive contact with reality?
Added: At the very least I can say that I am not going to make commitments to do specific things that violate my current ethics. I have certainly made no positive commitment to violate people’s bodily autonomy nor have such an intention.
Fair, I also haven’t made any specific commitments, I phrased it wrongly. I agree there can be extreme scenarios with trillions of digital minds tortured where you’d maybe want to declare war on the. rest of society. But I would still like people to write down that “of course, I wouldn’t want to destroy Earth before we can save all the people who want to live in their biological bodies, just to get a few years of acceleration in the cosmic conquest”. I feel a sentence like this should really have been included in the original post about dismantling the Sun, and until people are not willing to write this down, I remain paranoid that they would in fact haul the Amish the extermination camps if it feels like a good idea at the time. (As I said, I met people who really held this position.)