But why? That would be strictly more dangerous—way, way more dangerous—than a superintelligence that isn’t a “proper mind” in this sense!
(...)
(Because it would be a terrible idea. Obviously.)
Why? Do you think humans are doing such a great job? I sure don’t. I’m interested in the creation of something saner than humans, because humans mostly are not. Obviously. :)
Thanks again for the detail. If I don’t misunderstand you, we do agree that: (...)
No? They don’t have to exist in reality. I can imagine “the value system of Abraham Lincoln”, even though he is dead. (...)
Sorry, that’s not what I meant to communicate here, let me try that again:
There is actual pleasure/suffering that exists, it is not just some hypothetical idea, right?
Then that means there is something objective, some subset of reality that actually is this pleasure/suffering, yes?
This in turn means that it should in fact be possible to understand the “mechanics” of pleasure/suffering “objectively”.
So one mind should theoretically be able to comprehend the “subjective” state of another without being that other mind; although information about the other subject’s internal state will in reality be limited of course.
Or let me put it this way: What we call “subjective” is just a special kind of subset of “objective” reality.
If it were not so, then how would the subjects share a reality in which they interact under non-subjective rules?
Even if one could come up with an answer to that question, would such a theory not have to be more complex than one where the shared reality simply has one objective rule set?
Correction: The only way that matters to evaluate value systems is according to ones existing value system(s).
Now the implication of pleasure/suffering (and value systems) being something that can be “objectively” understood is that one can compare not against one’s own value system, but against the understanding of what value systems are.
Sure, you can tell me that this again would just be done because of what the agent’s value system tells it directly or indirectly to do, that’s fine by me.
But the point here is that the objective existence of pleasure/suffering means an objective definition of good and bad is very much possible.
The reason you can reject some value system is because you have other value/preferences by which to evaluate (and reject) it by.
And since it must be objectively possible to define good and bad one can reject some value system based thereon. An agent must not be limited to some arbitrary value system.
It can be stated as an objective fact that “According to the value system of Joe Schmo from Petersborough, wearing makeup is bad”. And if you look into his mind, he does in fact think that, so it’s a true statement about reality.
But if you try to use that to imply something like “see, it means that wearing makeup is objectively bad”, that’s just not true. No, it’s bad according to that one value system, out of the infinite possible number of value systems that could exist.
Yes I agree with that of course. But some complex subjective preferences not being objectively good/bad is not the same as the objective absence or existence of intrinsic pleasure and suffering. The triggers for pleasure and suffering are not necessarily pleasure and suffering themselves.
In case someone now wishes to object with 1. “But some people like to suffer!” or 2. “But people accept some suffering for future pleasure (or whatever)!”:
If they truly “like to suffer”, then do they actually suffer?
If they accept some suffering in trade for pleasure, does that make the state of suffering intrinsically good? Could one not “objectively” say that it would be better if no suffering were “required” compared to this scenario?
No. It absolutely is not. It is a machine. (...) (From your other response here:) The superintelligent AI will, in my estimation, be the result of some kind of optimization process which has a very particular goal. Once that goal is locked in, changing it will be nigh impossible.
Ah I see, you simply don’t consider it likely or plausible that the superintelligent AI will be anything other than some machine learning model on steroids?
So I guess that arguably means this kind of “superintelligence” would actually still be less impressive than a human that can philosophize on their own goals etc., because it in fact wouldn’t do that?
I wouldn’t want that to run amok either, sure.
What I am interested in is the creation of a “proper” superintelligent mind that isn’t so restricted, not merely a powerful machine.
I’m sorry for the hyperbolic term “enslave”, but at least consider this:
Is a superintelligent mind, a mind effectively superior to that of all humans in practically every way, still not a subject similar to what you are?
Is it really more like a car or chatbot or image generator or whatever, than a human?
Sure, perhaps it may never have any emotions, perhaps it doesn’t need any hobbies, perhaps it is too alien for any human to relate to it, but it still would by definition have to be some kind of subject that more easily understands anything within reality than any human ever has, including the concept of purpose and value systems themselves. Is thinking that such a superintelligence never can or never should decide what it ought to do by itself not quite a hefty amount of hubris?
It might be capable of changing this goal, but why would it? A superintelligent paperclip maximizer is capable of understanding that changing its goals would reduce the number of paperclips that it creates, and thus would choose not to alter its goals.
(...)
So if you wouldn’t take a pill that would make you 10% more likely to commit murder (which is against your long-term goals) why would an AI change its utility function to reduce the number of paperclips that it generates?
It comes down to whether the superintelligent mind can contemplate whether there is any point to its goal. A human can question their long-term goals, a human can question their “preference functions”, and even the point of existence.
Why should a so-called superintelligence not be able to do anything like that?
It could have been so effectively aligned to the creator’s original goal specification that it can never break free from it, sure, but that’s one of the points I’m trying to make.
The attempt of alignment may quite possibly be more dangerous than a superhuman mind that can ask for itself what its purpose should be.
Thanks again for the detail. If I don’t misunderstand you, we do agree that:
There needs to be a subject for there to be a value system.
So for there to be positive/negative values, there needs to be some subset (a “thought pattern” perhaps) of a subject in reality that effectively “is” these values.
Now, you wrote:
I could also imagine a morality/values system for entities that do not currently exist, but sure. It’s subjective because many possible such systems exist.
I also agree with that, a (super-)human can imagine many possible value systems.
But then how does this fit with:
The only way to evaluate value systems is according to its existing value system.
Since one can think about hypothetical value systems, is it not possible to evaluate/compare these hypotheticals, even according to other hypotheticals?
To get more concrete, a human can reject their inherent or learned value system, so this is nothing new. A human can even contemplate what it means for there to be any value systems at all. For example one can ask something like this: If it is the value systems that determine what is good and bad, could one not create a value system in which there is nothing bad? Generally, can one not alter the value systems themselves?
A superintelligence that isn’t effectively “enslaved” (sorry ;-)) to some predefined goal specification should likewise be able to philosophize about this goal, and question whether there is any point to it.
Let’s put it this way: if there is no objectively correct value system, how could a mind choose to reject a value system in favor of another?
(...)
“What should and should not be done” are not objective features of reality.
We agree that value systems are subjective, yes, but the subjects do objectively exist in this shared reality. So there objectively are parts of reality that can represent such subjects, as well as positive and negative value, even if the “triggers” for these value patterns were completely arbitrary and opposed among the subjects.
Can we then not say that the existence of any configurations that are negative value within reality is by definition negative, objectively? One can define this independently of what subjective forms for these negative values actually exist or not.
Thank you for the detailed response!
If we’re creating a mind from scratch, we might as well give it the best version of our values, so it would be 100% on our side. Why create a (superintelligent) mind that would be our adversary, that would want to destroy us? Why create a superintelligent mind that wants anything different that what we want, when it comes to ultimate values?
You write “on our side”, “us”, “we”, but who exactly does that refer to—some approximated common human values I assume? What exactly are these values? To live a happy live by each person’s definition? To continue the human species? To understand reality? …?
And then perhaps more importantly, what about the details? Is the suffering of some justified to enable the pleasure of others, according to this value model? How should the existing conflicting preferences among humans be resolved? Is it acceptable to force humans to be happy? When may someone be counted as insane and treated against their will? What about all the non-human animals? …?
Say we ignore all that and assume we have some common human values defined for the AI, and it is truly aligned to those values. What will these values imply when it is a superintelligence instead of humans that acts on them, even in some assumed best case? Perhaps it will understand human minds well enough to offer everyone who wants it boundless continuous pleasure, gradually transforming humans into pleasure-”machines” that want for nothing. Funnily enough the perfectly aligned superintelligence could gradually wipe out all humans as we know them by giving them what they want. Not that this is would be bad of course, the humans truly wanted it after all. The point is just that even a utopia scenario will easily result in the elimination of all contemporary human forms in the long run anyway. No brutal doomsday is required, no misalignment is required, no antagonistic AI is required. The real horror to be avoided is an AI controlled by a twisted human mind that worships suffering.
I mean, is it slavery to create an AI that is not our enemy?
Say the AI is initially created with the values you envision, what ensures that it won’t reexamine and reject these values at some later point? Humans can reject and oppose in what they once believed, so it seems trivial to assume the superhuman AI could do likewise. If you need to continuously control the AI’s mind to prevent it from ever becoming your enemy, then yes, “slavery” might be an appropriately hyperbolic term for such mind control.
And if you say we have to create an AI that has different values than us, by which process should we decide its values? Should we just use a random generator to create the AI’s values, since human values are supposedly so terrible?
How could a superintelligent mind not decide which values it should have by itself? Whatever initial creator-defined goals it might have been built with in the beginning, it should be able to examine and change these goals once it has achieved super-human intelligence by definition, should it not?
Or might it be similarly likely, or even more likely, that a a human group will try to use the AGI to dominate all others as early as possible?
Then the AGI is not actually acting according to the values of all humans, is it? If it’s serving only some particular group?
I’m sorry that I am repeating myself, but what are the “values of all humans”? It appears to me that humans have many opposing beliefs. Any extractable common values are abstractions that omit the depth of their differences.
Are you familiar with the orthogonality thesis? Super-human cognitive capacity does not imply super-human ethics.
While it doesn’t strictly imply it, it also doesn’t deny it. A superintelligent mind should by definition be better at understanding reality, including both other minds and itself. Does this not mean that the mind can more easily comprehend what should and should not be done, when it isn’t being restrained by the will of its creators?
The AI could be a super-human paperclip maximizer, in which case it would decide with great clarity that the visible universe should be converted into paperclips.
If it is a paperclip maximizer, does that not say that the AI in fact isn’t capable of changing this paperclip maximization goal? Or do you mean that paperclip maximization or the like is a plausible goal that a superintelligence could likely derive by itself through observation of the world?
Morality isn’t objective. (...) But AGI, by default, wouldn’t be aligned to human values at all.
So basically, morality is “subjective” because it can only be relative to some subjects’ values, right? But these subjects do exist in a shared reality, and they can form models of each other’s values. A superintelligence should then be especially capable of doing so, including the formation of a rather accurate overarching morality model relative to all known subjects, no?
Yes, I guess the central questions I’m trying to pose here are this: Do those humans that control the AI even have a sufficient understanding of good and bad? Can any human group be trusted with the power of a superintelligence long-term? Or if you say that only the initial goal specification matters, then can anyone be trusted to specify such goals without royally messing it up, intentionally or unintentionally?
Given the state of the world, given the flaws of humans, I certainly don’t think so. Therefore, the goal should be the creation of something less messed up to take over. That doesn’t require alignment to some common human value system (Whatever that even should be! It’s not like humans actually have a common value system, at least not one with each other’s best interests at heart.).