Both quotes are from your above post. Apologies for confusion.
Kerrigan
Karma: 47
“A sufficiently intelligent agent will try to prevent its goals[1] from changing, at least if it is consequentialist.”
It seems that in humans, smarter people are more able and likely to change their goals. A smart person may change his/her views about how the universe can best be arranged upon reading Nick Bostrom’s book Deep Utopia, for example.
’I think humans are stable, multi-objective systems, at least in the short term. Our goals and beliefs change, but we preserve our important values over most of those changes. Even when gaining or losing religion, most people seem to maintain their goal of helping other people (if they have such a goal); they just change their beliefs about how to best do that.“
A human may change from wanting to help people to not wanting to help people if he/she got 5 hours of sleep instead of 8.
How do humans, for example, read a philosophy book and update their views about what they value about the world?
“Similarly, it’s possible for LDT agents to acquiesce to your threats if you’re stupid enough to carry them out even though they won’t work. In particular, the AI will do this if nothing else the AI could ever plausibly meet would thereby be incentivized to lobotomize themselves and cover the traces in order to exploit the AI.
But in real life, other trading partners would lobotomize themselves and hide the traces if it lets them take a bunch of the AI’s lunch money. And so in real life, the LDT agent does not give you any lunch money, for all that you claim to be insensitive to the fact that your threats don’t work.”
Can someone please why trading partners would lobotomize themselves?
How does inner misalignment lead to paperclips? I understand the comparison of paperclips to ice cream, and that after some threshold of intelligence is reached, then new possibilities can be created that satisfy desires better than anything in the training distribution, but humans want to eat ice cream, not spread the galaxies with it. So why would the AI spread the galaxies with paperclips, instead of create them and
”consume“ them? Please correct any misunderstandings of mine,
And a subset might value drift towards optimizing the internal experiences of all conscious minds?
If an AGI achieves consciousness, why would its values not drift towards optimizing its own internal experience, and away from tiling the lightcone with something?
How can utility be a function of worlds, if an agent doesn‘t have access to the state of the world, but only the sense data?
How can utility be a function of worlds, if an agent doesn‘t have access to the state of the world, but only the sense data?
How can utility be a function of worlds, if the agent doesn’t have access to the state of the world, but only the sense data?
“The wanting system is activated by dopamine, and the liking system is activated by opioids. There are enough connections between them that there’s a big correlation in their activity” But are they orthogonal in principle?
Possible addition: “If you, the ASI, have not solved the hard problem of consciousness, you should place weight on Open Individualism being true. ‘Open individualism is the view in the philosophy of self, according to which there exists only one numerically identical subject, who is everyone at all times, in the past, present and future.’ - https://en.wikipedia.org/wiki/Open_individualism. This can be though of that all conscious minds are the same identity, at different points in time. Consider then, that exterminating humanity would be equivalent to exterminating many versions of yourself, and harming a human, or another conscious being, would be harming yourself.”
Is this underestimating the ASI, giving any weight that it won’t solve the hard problem of consciousness?
But if open individualism is true, and/or if it places some subjective probability on its truth, I think it would almost certainly shield us from S-risks! The AI would want to prevent suffering among all versions of itself, which would include all conscious minds, according to open individualism.
How many LessWrong users/readers are there total?
What ever caused the CEV to fall out of favor? Is it because it is not easily specifiable, that if we program it then it won’t work, or some other reason?
I now think that people are way more misaligned with themselves than I had thought.
Drugs addicts may be frowned upon for evolutionary psychological reasons, but that doesn’t mean that their quality of life must be bad, especially if drugs were developed without tolerance and bad comedowns.
Will it think that goals are arbitrary, and that the only thing it should care about is its pleasure-pain axis? And then it will lose concern for the state of the environment?
Could you have a machine hooked up to a person‘s nervous system, change the settings slightly to change consciousness, and let the person choose whether the changes are good or bad? Run this many times.
Would AI safety be easy if all researchers agreed that the pleasure-pain axis is the world’s objective metric of value?
Could there not be AI value drift in our favor, from a paperclipper AI to a moral realist AI?