Just want to articulate one possibility of how the future could look like:
RL agents will be sooo misaligned so early, they would lie and cheat and scheme all the time, so that alignment becomes a practical issue, with normal incentives, and get iteratively solved for not-very-superhuman agents. Turns out it requires mild conceptual breakthroughs, as these agents are slightly superhuman, fast, and hard to supervise directly to just train away the adversarial behaviors in the dumbest way possible. It finishes developing by the time of ASI arrival and people just align it with a lot of effort, in the same manner as any big project requires a lot of effort.
I’m not saying anything about the probability of it. It honestly feels a bit overfitted, just like people who overupdated on base models talked for some time. But still, the whole LLM arc was kind of weird and goofy, so I don’t trust my sense of weird and goofy anymore.
(would appreciate references of forecasting writeups exploring similar scenario)
Suppose you are writing a simulation. You keep optimizing it and hardcoding some stuff and handle different cases more efficiently and everything. One day your simulation becomes efficient enough so that you can run big enough grid for long enough, and there develops life. Then intelligent life. Then they tried to figure out the physics of their universe, and they succeed! But, oh wait, their description is extremely short but completely computationally intractable.
Can you say that they actually figured out in what kind of universe they are in already, or should you wait for when they discover another million lines of code of optimization for it? Should you create giant sparkling letters “congratulations, you figured that out” or wait for more efficient formulation?
The convergent reason to simulate a world is to learn what happens there. When to intervene with letters depends on, uh. Why are you doing that at all?
(Edit: I suppose a congratulatory party is in order when they simulate you back with enough optimizations that you can talk to each other in real time using your mutual read access.)
I would really love if some “let’s make asi” people put some effort into making bad outcomes less bad. Like, it would really suck if we are going to be trapped in endless corporate punk hell, with superintelligent nannies with correct (tm) opinions. Or infinite wedding parties or whatever. Just make sure that if you fuck up we all just get eaten by nanobots please. Permanent entrapment in misery would be a lot worse.
Suppose you know that there is an apple in this box. You will modify your memory then, to think that the box is empty. You open the box, expecting nothing there. Is there an apple?
Also, what if there is another branch of the universe where there is no apple, and you in the “yes apple” universe did modify his memory and you are both identical now. So there are two identical people in different worlds, one with box-with-apple, the other one with box-without-apple.
Should you, in the world with apple and yet unmodified memory anticipate 50% chance to experience empty box after opening it?
I think it’s identical to the problem when you get copied in two rooms, numbered 1 and 2, then you should expect 50% of 1 and 50% of 2 even if there is literally no randomness or uncertainty in what’s going to happen. or is it?
So, implication’s here is that you can squeeze yourself into different timelines by modifying your memory or what, am i going crazy here
In our solar system, the two largest objects are the Sun and Jupiter. Suspiciously, their radii both start with the number ’69′: the Sun’s radius is 696,340 km, while Jupiter’s is 69,911 km.
What percent of ancestral simulations have this or similarly silly “easter eggs”. What is the Bayes factor
TLDR give pigs guns (preferably by enhancing individual baseline pigs, not by breeding new type of smart powerful pig. Otherwise it will probably just be two different cases. More like gene therapy than producing modified fetuses)
As of lately I hold an opinion that morals are proxy to negotiated cooperation or something, I think it clarifies a lot about the dynamics that produce it. It’s like evolutionary selection → human desire to care about family and see their kids prosper, implicit coordination problems between agents of varied power levels → morals.
So, like, uplift could be the best way to ensure that animals are treated well. Just give them power to hurt you and benefit you, and they will be included into moral considerations, after some time for it to shake out. Same stuff with hypothetical p-zombies, they are as powerful as humans, so they will be included. Same with EMs.
Also, “super beneficiaries” are then just powerful beings, don’t bother to research the depth of experience or strength of preferences. (e.g. gods, who can do whatever and don’t abide by their own rules and perceived to be moral, as an example of this dynamics).
Also, pantheon of more human like gods → less perceived power + perceived possibility to play on disagreements → lesser moral status. One powerful god → more perceived power → stronger moral status. Coincidence? I think not.
Modern morals could be driven by a lot stronger social mobility. People have a lot of power now, and can unexpectedly acquire a lot of power later. so, you should be careful with them and visibly commit to treating them well (e.g. be moral person, with particular appropriate type of morals).
And it’s not surprising how (chattel) slaves were denied a claim on being provided with moral considerations (or claim on being a person or whatever), in a strong equilibrium where they are powerless and expected to be powerless later.
tldr give pigs guns
(preferably by enhancing individual baseline pigs, not by breeding new type of smart powerful pig. Otherwise it will probably just be two different cases. More like gene therapy than producing modified fetuses)
I think this is misguided. It ignores the is-ought discrepancy by assuming that the way morals seem to have evolved is the “truth” of moral reasoning. I also think it’s tactically unsound—the most common human-group reaction to something that looks like a threat and isn’t already powerful enough to hurt us is extermination.
I DO think that uplift (of humans and pigs) is a good thing on its own—more intelligence means more of the universe experiencing and modeling itself.
It ignores the is-ought discrepancy by assuming that the way morals seem to have evolved is the “truth” of moral reasoning
No? Not sure how do you got that from my post. Like, my point is that morals are baked in solutions to coordination problems between agents with different wants and power levels. Baked into people’s goal systems. Just as “loving your kids” is a desire that was baked in from reproductive fitness pressure. But instead of brains it works on a level of culture. I.e. Adaptation-Executers, not Fitness-Maximizers
I also think it’s tactically unsound—the most common human-group reaction to something that looks like a threat and isn’t already powerful enough to hurt us is extermination.
Eh. I think it’s one of the considerations. Like, it will probably not be that. It’s either ban on everything even remotely related or some chaos when different regulatory systems trying to do stuff.
Just want to articulate one possibility of how the future could look like:
RL agents will be sooo misaligned so early, they would lie and cheat and scheme all the time, so that alignment becomes a practical issue, with normal incentives, and get iteratively solved for not-very-superhuman agents. Turns out it requires mild conceptual breakthroughs, as these agents are slightly superhuman, fast, and hard to supervise directly to just train away the adversarial behaviors in the dumbest way possible. It finishes developing by the time of ASI arrival and people just align it with a lot of effort, in the same manner as any big project requires a lot of effort.
I’m not saying anything about the probability of it. It honestly feels a bit overfitted, just like people who overupdated on base models talked for some time. But still, the whole LLM arc was kind of weird and goofy, so I don’t trust my sense of weird and goofy anymore.
(would appreciate references of forecasting writeups exploring similar scenario)
Suppose you are writing a simulation. You keep optimizing it and hardcoding some stuff and handle different cases more efficiently and everything. One day your simulation becomes efficient enough so that you can run big enough grid for long enough, and there develops life. Then intelligent life. Then they tried to figure out the physics of their universe, and they succeed! But, oh wait, their description is extremely short but completely computationally intractable.
Can you say that they actually figured out in what kind of universe they are in already, or should you wait for when they discover another million lines of code of optimization for it? Should you create giant sparkling letters “congratulations, you figured that out” or wait for more efficient formulation?
The convergent reason to simulate a world is to learn what happens there. When to intervene with letters depends on, uh. Why are you doing that at all?
(Edit: I suppose a congratulatory party is in order when they simulate you back with enough optimizations that you can talk to each other in real time using your mutual read access.)
I would really love if some “let’s make asi” people put some effort into making bad outcomes less bad. Like, it would really suck if we are going to be trapped in endless corporate punk hell, with superintelligent nannies with correct (tm) opinions. Or infinite wedding parties or whatever. Just make sure that if you fuck up we all just get eaten by nanobots please. Permanent entrapment in misery would be a lot worse.
Suppose you know that there is an apple in this box. You will modify your memory then, to think that the box is empty. You open the box, expecting nothing there. Is there an apple?
Also, what if there is another branch of the universe where there is no apple, and you in the “yes apple” universe did modify his memory and you are both identical now. So there are two identical people in different worlds, one with box-with-apple, the other one with box-without-apple.
Should you, in the world with apple and yet unmodified memory anticipate 50% chance to experience empty box after opening it?
If you got confused about the setup here is a diagram: https://i.imgur.com/jfzEknZ.jpeg
I think it’s identical to the problem when you get copied in two rooms, numbered 1 and 2, then you should expect 50% of 1 and 50% of 2 even if there is literally no randomness or uncertainty in what’s going to happen. or is it?
So, implication’s here is that you can squeeze yourself into different timelines by modifying your memory or what, am i going crazy here
In our solar system, the two largest objects are the Sun and Jupiter. Suspiciously, their radii both start with the number ’69′: the Sun’s radius is 696,340 km, while Jupiter’s is 69,911 km.
What percent of ancestral simulations have this or similarly silly “easter eggs”. What is the Bayes factor
You might enjoy this classic: https://www.lesswrong.com/posts/9HSwh2mE3tX6xvZ2W/the-pyramid-and-the-garden
TLDR give pigs guns (preferably by enhancing individual baseline pigs, not by breeding new type of smart powerful pig. Otherwise it will probably just be two different cases. More like gene therapy than producing modified fetuses)
As of lately I hold an opinion that morals are proxy to negotiated cooperation or something, I think it clarifies a lot about the dynamics that produce it. It’s like evolutionary selection → human desire to care about family and see their kids prosper, implicit coordination problems between agents of varied power levels → morals.
So, like, uplift could be the best way to ensure that animals are treated well. Just give them power to hurt you and benefit you, and they will be included into moral considerations, after some time for it to shake out. Same stuff with hypothetical p-zombies, they are as powerful as humans, so they will be included. Same with EMs.
Also, “super beneficiaries” are then just powerful beings, don’t bother to research the depth of experience or strength of preferences. (e.g. gods, who can do whatever and don’t abide by their own rules and perceived to be moral, as an example of this dynamics).
Also, pantheon of more human like gods → less perceived power + perceived possibility to play on disagreements → lesser moral status. One powerful god → more perceived power → stronger moral status. Coincidence? I think not.
Modern morals could be driven by a lot stronger social mobility. People have a lot of power now, and can unexpectedly acquire a lot of power later. so, you should be careful with them and visibly commit to treating them well (e.g. be moral person, with particular appropriate type of morals).
And it’s not surprising how (chattel) slaves were denied a claim on being provided with moral considerations (or claim on being a person or whatever), in a strong equilibrium where they are powerless and expected to be powerless later.
tldr give pigs guns
(preferably by enhancing individual baseline pigs, not by breeding new type of smart powerful pig. Otherwise it will probably just be two different cases. More like gene therapy than producing modified fetuses)
I think this is misguided. It ignores the is-ought discrepancy by assuming that the way morals seem to have evolved is the “truth” of moral reasoning. I also think it’s tactically unsound—the most common human-group reaction to something that looks like a threat and isn’t already powerful enough to hurt us is extermination.
I DO think that uplift (of humans and pigs) is a good thing on its own—more intelligence means more of the universe experiencing and modeling itself.
No? Not sure how do you got that from my post. Like, my point is that morals are baked in solutions to coordination problems between agents with different wants and power levels. Baked into people’s goal systems. Just as “loving your kids” is a desire that was baked in from reproductive fitness pressure. But instead of brains it works on a level of culture. I.e. Adaptation-Executers, not Fitness-Maximizers
Eh. I think it’s one of the considerations. Like, it will probably not be that. It’s either ban on everything even remotely related or some chaos when different regulatory systems trying to do stuff.