“For there to be a concept, there has to be a boundary. So what am I recognizing?”
I think you’re just recognizing that the alien artifact looks like something that wouldn’t occur naturally on Earth, rather than seeing any kind of essence. Because Earth is where we originally made the concept, and we didn’t need an essence there, we just divided the things we know we made from the things we know we didn’t.
I think I see where the disconnect was in this conversation. Lanier was accusing general AI people of being religious. Yudkowsky took that as a claim that something he believed was false, and wanted Lanier to say what.
But Lanier wasn’t saying anything in particular was false. He was saying that when you tackle these Big Problems, there are necessarily a lot of unknowns, and when you have too many unknowns reason and science are inapplicable. Science and reason work best when you have one unknown and lots of knowns. If you try to bite off too big a chunk at once you end up reasoning in a domain that is now only e.g. 50% fact, and that reminds him of the “reasoning” of religious people.
Knowledge is a big interconnected web, with each fact reinforcing the others. You have to grow it from the edge. And our techniques are design for edge space.
“Ayn Rand? Aleister Crowley? How exactly do you get there? What Rubicons do you cross? It’s not the justifications I’m interested in, but the critical moments of thought.”
Do you want to know the critical moments of thought so you can shape an AI that will not have them? I think Ayn Rand’s key realization was that concepts can have interdependencies.
For example, if you don’t already have the concept “property” then no matter how many concrete examples of stealing you see, you won’t be able to form the concept “stealing.” You won’t be able to “see” what they’re doing without that earlier knowledge, you will just see them as people moving stuff around.
Then at some point she realized the same kind of dependency relationship exists between “value” and “life,” and that it implies something about what the standard of value must be.
But then, worse case, if an AI did discover this, and follow the same path, it would not become a selfish psychopath. It would conclude that since it is not alive, the concept “value” does not apply to it. Therefore the only reason to perform any act at all is if it helps us, the living.
The ability to become emotionally detached is a useful skill (e.g. if you are being tortured) but when it becomes an automatic reflex to any emotion, it can take all the colour out of life.
Sometimes highly intelligent people are also overwhelmingly sensitive/empathetic so detaching is very tempting. The first few minutes of this video with the genius girl walking around the spaceship shows what it’s like to be highly empathetic (Firefly).
http://www.youtube.com/watch?v=MsyuTLYx59g
But also: emotions come from the subconscious, and the subconscious contains that which is done repeatedly on the conscious level. So if you are habitually rational, does that effect your subconscious and therefore your emotions?
I think what happens is, you are so consistent on the conscious level (e.g. the way the our host cross-links all his posts) that the subconscious is also highly consistent. So when it produces an emotion it produces it with the whole of itself, instead of just one part contradicted by another (mixed emotions). Therefore the genius has very strong emotions, which interestingly is the stereotype: the overwrought genius who flys off the handle.
The sheer strength of having a conscious and subconscious in total agreement, and both in turn in agreement with reality, can be overwhelming and, like the girl above (“It’s getting very crowded in here!”) you just want to shut it off.
I agree that there are certain moral rules we should never break. Human beings are not omniscient, so all of our principles have to be principles-in-a-context. In that sense every principle is vulnerable to a black swan, but there are levels of vulnerability. The levels correspond to how wide ranging the abstraction. The more abstract the less vulnerable.
Injunctions about truth are based on the metaphysical fact of identity, which is implied in every single object we encounter in our entire lives. So epistemological injunctions are the most invulnerable. The one about not helping the ferry boat captain—well helping him would be an absolute in normal life, but war is not normal life. It’s a big, ugly, black swan. They should not feel guilty over that poor fellow, because “it’s just war.” (and I mean that in a deep epistemological sense, not a redneck sense)
I could not live under a non-capitalist system. I believe people should be able to keep what they earn and charity should be a private affair.
I don’t think it’s possible that our hardware could trick us in this way (making us doing self-interested things by making them appear moral).
To express the idea “this would be good for the tribe” would require the use of abstract concepts (tribe, good) but abstract concepts/sentences are precisely the things that are observably under our conscious control. What can pop up without our willing it are feelings or image associations so the best trickery our hardware could hope for is to make something feel good.
The meta argument others have mentioned—“Telling the world you let me out is the responsible thing to do,” would work on me.
Re: why rationality can’t be learned by rote -
If you introspect on a process of reason, you see that you actually choose at each step which path of inquiry to follow next and which to ignore. Each choice takes the argument to the next step, ultimately driving it to completion. Reason is “powered by choice(TM)” which is why it is incoherent to argue rationally for determinism and also why it can’t be learned by rote.
Software developers (such as myself) in our more abstract moments can think of reason as simply encoding ones premises as a string of symbols standing for definitions and mechanically applying the rules of deduction (Prolog style). But introspection belies this—it’s actually highly creative and messy. Reason is an art not a science.
Except the universe doesn’t care how much backbreaking effort you make, only if you get the cause and effect right. Which is why cultures that emphasize hard work are not overtaking cultures that emphasize reason (Enlightenment cultures). Of course even these cultures must still do some work, that of enacting their cleverly thought out causes.
“If you are talking about a timescale of decades, than intelligence augmentation does seems like a worthy avenue of investment”
This seems like the way to go to me. It’s like “generation ships” in sci-fi. Should we launch ships to distant star systems today, knowing that ships launched 10 years from now will overtake them on the way?
Of course in the case of AI, we don’t know what the rate of human enhancement will be, and maybe the star isn’t so distant after all.
I don’t want to sign up for cryonics because I’m afraid I will be revived brain-damaged. But maybe others are worried they will have the social status of a freak in that future society.
I don’t believe IQ tests measure everything. There’s a certain feeling when being creative, and when completing these tests I have not felt it, so I don’t think it’s measuring it.
Also I am not sure intelligence is general. At the level of ordinary life it certainly is, but geniuses are always geniuses at something, e.g. maths, physics, composing. Why aren’t they geniuses at everything.
Reminds me of this: “Listen, and understand. That terminator is out there. It can’t be bargained with. It can’t be reasoned with. It doesn’t feel pity, or remorse, or fear. And it absolutely will not stop, ever, until you are dead.”
But my question would be: Is the universe of cause and effect really so less safe than the universe of God? At least in this universe, someone who has an evil whim is limited by the laws of cause and effect, e.g. Hitler had to build tanks first, which gave the allies time to prepare. In that other universe, Supreme Being decides he’s bored with us and zap we’re gone, no rules he has to follow to achieve that outcome.
So why is relying on the goodness of God safer than relying on the inexorability of cause and effect?
When Yoda said “there is no try,” I took it more literally. In the absence of human concepts there is no “try,” there is only things that act or don’t act. Let go of your mind and all that.
“I understood that you could do everything that you were supposed to do, and Nature was still allowed to kill you.”
You finally realized inanimate objects can’t be negotiated with… and then continued with your attempt to rectify this obvious flaw in the universe :)
One theory has a track record of prediction, and what is being asked for is a prediction, so at first glance I would choose that one. But the explanation based-one is built on more data.
But it is neither prediction nor explanation that makes things happen in the real world, but causality. So I would look in to the two theories and pick the one that looks to have identified a real cause instead of simply identifying a statistical pattern in the data.
The observations in this post gel with my experience also.
Middle managers can be the most short-sided, penny-pinching, over-simplifying people in the world. But when you talk to CEOs they are often well-spoken, well-read, philosophical, long-term.
You ask them a business question and expect to get back balance sheets, dollars, etc. but instead you get something surprisingly wide-ranging/philosophical.
How can you tell if someone is an idiot not worth refuting, or if they’re a genius who’s so far ahead of you to sound crazy to you? Could we think an AI had gone mad, and reboot it, when it is really genius.
Earlier I said we are seeing things that are like what we make. But that’s not a very useful definition implementation-wise.
My own approach to implementation is to define intelligence as the results of a particular act—“thinking”—and then introspect to see what the individual elements of that act are, and implement them individually.
Yes, I went to Uni and was told intelligence was search, and all my little Prolog programs worked, but I think they were oversimplifying. They were unacknowledged Platonists, trying to find the hidden essence, trying to read God’s mind, instead of simply looking at what is (albeit through introspection) and attempting to implement it.
All very naive and 1800s of me, I know. Imagine using introspection! What an unthinkable cad. Well pardon me for actually looking at the thing I’m trying to program.