If we are too stupid, we are doomed anyway. If we become dramatically smarter than we are now, we will update all our moral theories anyway. On the level where we are now, following a few rules about not abusing power, which have a good outside-view support, seems like a good idea.
I agree with the part that if we become much smarter, it may be safe to do things that are dangerous for us now. What I doubt is the reliability of determining how much smarter we have actually became, if there is a strong temptation to scream: “yes, I am already much smarter, now shut up and let me take all those cookies!!!”
Given what I know about people, if we today agree that—hypothetically, in a distant posthuman future—it will be a good idea to take all power you can and forcefully optimize the universe for greater good… a few weeks later, after a few more articles elaborating this topic, some people will start publicly declaring that they are already that smart (and if you are not, that only means that you are a loser, and should obey them).
If we are too stupid, we are doomed anyway. If we become dramatically smarter than we are now, we will update all our moral theories anyway. On the level where we are now, following a few rules about not abusing power, which have a good outside-view support, seems like a good idea.
I agree with the part that if we become much smarter, it may be safe to do things that are dangerous for us now. What I doubt is the reliability of determining how much smarter we have actually became, if there is a strong temptation to scream: “yes, I am already much smarter, now shut up and let me take all those cookies!!!”
Given what I know about people, if we today agree that—hypothetically, in a distant posthuman future—it will be a good idea to take all power you can and forcefully optimize the universe for greater good… a few weeks later, after a few more articles elaborating this topic, some people will start publicly declaring that they are already that smart (and if you are not, that only means that you are a loser, and should obey them).