Independent AGI system builder. I came here as part of my search to find out what other NGI systems are doing to provide moral control of their AGI systems, but this is clearly not the right place for that. I will continue my search for intelligent life on Earth elsewhere.
David Cooper
Karma: −83
I wouldn’t want to try to program a self-less AGI system to be selfish. Honesty is a much safer route: not trying to build a system that believes things that aren’t true (and it would have to believe it has a self to be selfish). What happens if such deceived AGI learns the truth while you rely on it being fooled to function correctly? We’re trying to build systems more intelligent than people, don’t forget, so it isn’t going to be fooled by monkeys for very long.
Freezing programs contain serious bugs. We can’t trust a system with any bugs if it’s going to run the world. Hardware bugs can’t necessarily be avoided, but if multiple copies of an AGI system all work on the same problems and compare notes before action is taken, such errors can be identified and any affected conclusions can be thrown out. Ideally, a set of independently-designed AGI systems would work on all problems in this way, and any differences in the answers they generate would reveal faults in the way one or more of them are programmed. AGI will become a benign dictator—to go against its advice would be immoral and harmful, so we’ll soon learn to trust it.
The idea of having people vote faulty “AGI” into power from time to time isn’t a good one—there is no justification for switching between doing moral and immoral things for several years at a time.
This file looks spreadsheety --> it’s got lots of boxy fields
That wordprocessor is spreadsheety --> it can carry out computations on elements
(Compound property with different components of that compound property being referred to in different contexts.)
A spreadsheet is a combination of many functionalities. What is its relevance to this subject? It’s been brought in to suggest that properties like “spreadsheety” can exist without having any trace in the components, but no—this compound property very clearly consists of components. It’s even clearer when you write the software and find that you have to build it out of components. The pattern in which the elements are brought together is an abstract component, and abstract components have no substance. When we’re dealing with sentience and looking for something to experience pain, relying on this kind of component to perform that role is more than a little fanciful. Even if we make such a leap of the imagination though and have sentient geometries, we still don’t have a model as to how this experience of pain (or any other kind of feeling) can transfer to the generation of data which documents that experience.
“How do you know it exists, if science knows nothing about it?”
All science has to go on is the data that people produce which makes claims about sentience, but that data can’t necessarily be trusted. Beyond that, all we have is internal belief that the feelings we imagine we experience are real because they feel real, and it’s hard to see how we could be fooled if we don’t exist to be fooled. But an AGI scientist won’t be satisfied by our claims—it could write off the whole idea as the ramblings of natural general stupidity systems.
“This same argument applies just as well to any distributed property. I agree that intelligence/sentience/etc. does not arise from complexity alone, but it is a distributed process and you will not find a single atom of Consciousness anywhere in your brain.”
That isn’t good enough. If pain is experienced by something, that something cannot be in a compound of any kind with none of the components feeling any of it. A distribution cannot suffer.
“Is your sentience in any way connected to what you say?”
It’s completely tied to what I say. The main problem is that other people tend to misinterpret what they read by mixing other ideas into it as a short cut to understanding.
“Then sentience must either be a physical process, or capable of reaching in and pushing around atoms to make your neurons fire to make your lips say something. The latter is far more unlikely and not supported by any evidence. Perhaps you are not your thoughts and memories alone, but what else is there for “you” to be made of?”
Focus on the data generation. It takes physical processes to drive that generation, and rules are being applied in the data system to do this with each part of that process being governed by physical processes. For data to be produced that makes claims about experiences of pain, a rational process with causes and effects at every step has to run through. If the “pain” is nothing more than assertions that the data system is programmed to churn out without looking for proof of the existence of pain, there is no reason to take those assertions at face value, but if they are true, they have to fit into the cause-and-effect chain of mechanism somewhere—they have to be involved in a physical interaction, because without it, they cannot have a role in generating the data that supposedly tells us about them.
“So the Sentiences are truly epiphenomenonological, then? (They have no causal effect on physical reality?) Then how can they be said to exist? Regardless of the Deep Philosophical Issues, how could you have any evidence of their existence, or what they are like?”
Repeatedly switching the sentient thing wouldn’t remove its causal role, and nor would having more than one sentience all acting at once—they could collectively have an input even if they aren’t all “voting the same way”, and they aren’t going to find out if they got their wish or not because they’ll be loaded with a feeling of satisfaction that they “won the vote” even if they didn’t, and they won’t remember which way they “voted” or what they were even “voting” on.
“They are both categories of things.”
“Chairness” is quite unlike sentience. “Chairness” is an imagined property, whereas sentience is an experience of a feeling.
“It’s the same analogy as before—just as you don’t need to split a chair’s atoms to split the chair itself, you don’t need to make a brain’s atoms suffer to make it suffer.”
You can damage a chair with an axe without breaking every bond, but some bonds will be broken. You can’t split it without breaking any bonds. Most of the chair is not broken (unless you’ve broken most of the bonds). For suffering in a brain, it isn’t necessarily atoms that suffer, but if the suffering is real, something must suffer, and if it isn’t the atoms, it must be something else. It isn’t good enough to say that it’s a plurality of atoms or an arrangement of atoms that suffers without any of the atoms feeling anything, because you’ve failed to identify the sufferer. No arrangement of non-suffering components can provide everything that’s required to support suffering.
″ “Nothing is ever more than the sum of its parts (including any medium on which it depends). Complex systems can reveal hidden aspects of their components, but those aspects are always there.” --> How do you know that? And how can this survive contact with reality, where in practice we call things “chairs” even if there is no chair-ness in its atoms?”
“Chair” is a label representing a compound object. Calling it a chair doesn’t magically make it more than the sum of its parts. Chairs provide two services—one that they support a person sitting on them, and the other that they support someone’s back leaning against it. That is what a chair is. You can make a chair in many ways, such as by cutting out a cuboid of rock from a cliff face. You could potentially make a chair using force fields. “Chairness” is a compound property which refers to the functionalities of a chair. (Some kinds of “chairness” could also refer to other aspects of some chairs, such as their common shapes, but they are not universal.) The fundamental functionalities of chairs are found in the forces between the component atoms. The forces are present in a single atom even when it has no other atom to interact with. There is never a case where anything is more than the sum of its parts—any proposed example of such a thing is wrong.
“I recommend the Reductionism subsequence.”
Is there an example of something being more than the sum of its parts there? If so, why don’t we go directly to that. Give me your best example of this magical phenomenon.
“But the capability of an arrangement of atoms to compute 2+2 is not inside the atoms themselves. And anyway, this supposed “hidden property” is nothing more than the fact that the electron produces an electric field pointed toward it. Repelling-each-other is a behavior that two electrons do because of this electric field, and there’s no inherent “repelling electrons” property inside the electron itself.”
In both cases, you’re using compound properties where they are built up of component properties, and then you’re wrongly considering your compound properties to be fundamental ones.
“But it’s not a thing! It’s not an object, it’s a process, and there’s no reason to expect the process to keep going somewhere else when its physical substrate fails.”
You can’t make a process suffer.
“Taking the converse does not preserve truth. All cats are mammals but not all mammals are cats.”
Claiming that a pattern can suffer is a way-out claim. Maybe the universe is that weird though, but it’s worth spelling out clearly what it is you’re attributing sentience to. If you’re happy with the idea of a pattern experiencing pain, then patterns become remarkable things. (I’d rather look for something of more substance rather than a mere arrangement, but it leaves us both with the bigger problem of how that sentience can make its existence known to a data system.)
“You could torture the software, if it were self-aware and had a utility function.”
Torturing software is like trying to torture the text in an ebook.
“But—where is the physical sufferer inside you?”
That’s what I want to know.
“You have pointed to several non-suffering patterns, but you could just as easily do the same if sentience was a process but an uncommon one. (Bayes!)”
Do you seriously imagine that there’s any magic pattern that can feel pain, such as a pattern of activity where none of the component actions feel anything?
“There is already an explanation. There is no need to invoke the unobservable.”
If you can’t identify anything that’s suffering, you don’t have an explanation, and if you can’t identify how your imagined-to-be-suffering process or pattern is transmitting knowledge of that suffering to the processes that build the data that documents the experience of suffering, again you don’t have an explanation.
“What do you mean, it works? I agree that it matches our existing preconceptions and intuitions about morality better than the average random moral system, but I don’t think that that comparison is a useful way of getting to truth and meaningful categories.”
It works beautifully. People have claimed it’s wrong, but they can’t point to any evidence for that. We urgently need a system for governing how AGI calculates morality, and I’ve proposed a way of doing so. I came here to see what your best system is, but you don’t appear to have made any selection at all—there is no league table of best proposed solutions, and there are no league tables for each entry listing the worst problems with them. I’ve waded through a lot of stuff and have found that the biggest objection to utilitarianism is a false paradox. Why should you be taken seriously at all when you’ve failed to find that out for yourselves?
“You have constructed a false dilemma. It is quite possible for both of you to be wrong.”
If you trace this back to the argument in question, it’s about equal amounts of suffering being equally bad for sentiences in different species. If they are equal amounts, they are necessarily equally bad—if they weren’t, they wouldn’t have equal values.
″ “You’ve taken that out of context—I made no claim about it making moral judgements on the basis of intelligence alone. That bit about using intelligence alone was referring to a specific argument that doesn’t relate directly to morality.” --> ” “All sentiences are equally important” is definitely a moral statement.”
Again you’re trawling up something that my statement about using intelligence alone for was not referring to.
“First, it is not trivial to define what beings are sentient and what counts as suffering (and how much).”
That doesn’t matter—we can still aim to do the job as well as it can be done based on the knowledge that is available, and the odds are that that will be better than not attempting to do so.
″ Second, if your morality flows entirely from logic, then all of the disagreement or possibility for being incorrect is inside “you did the logic incorrectly,” and I’m not sure that your method of testing moral theories takes that possibility into account.”
It will be possible with AGI to have it run multiple models of morality and to show up the differences between them and to prove that it is doing the logic correctly. At that point, it will be easier to reveal the real faults rather than imaginary ones. But it would be better if we could prime AGI with the best candidate first, before it has the opportunity to start offering advice to powerful people.
“I agree that it is mathematics, but where is this “proper” coming from?”
Proper simply means correct—fair share where everyone gets the same amount of reward for the same amount of suffering.
“Could somebody disagree about whether, say, it is moral to harm somebody as retributive justice? Then the equations need our value system as input, and the results are no longer entirely objective.”
Retributive justice is inherently a bad idea because there’s no such thing as free will—bad people are not to blame for being the way they are. However, there is a need to deter others( and to discourage repeat behaviour by the same individual if they’re ever to be released into the wild again), so plenty of harm will typically be on the agenda anyway if the calculation is that this will reduce harm.
“Perhaps the reason that we disagree with you is not that we’re emotionally biased, irrational, mobbish, etc. Maybe we simply disagree. People can legitimately disagree without one of them being Bad People.”
It’s obvious what’s going on when you look at the high positive scores being given to really poor comments.
“It tells me that you missed the point. Parfit’s paradox is not about pragmatic decision making, it is about flaws in the utility function.”
A false paradox tells you nothing about flaws in the utility function—it simply tells you that people who apply it in a slapdash manner get the wrong answers out of it and that the fault lies with them.
“You have indeed found A Reason that supports your belief in the AGI-God, but I think you’ve failed to think it through. Why should the AGI need to tell us how we did in order to analyze our thought processes? And how come the optimal study method is specifically the one which allows you to be shown Right All Along? Specificity only brings Burdensome Details.”
AGI won’t be programmed to find me right all the time, but to identify which arguments are right. And for the sake of those who are wrong, they need to be told that they were wrong so that they understand that they are at reasoning and not the great thinkers they imagine themselves to be.
“In this scenario, it’s not gone, it’s never been to begin with.”
Only if there is no such thing as sentience, and if there’s no such thing, there is no “I” in the “machine”.
“I think that a sufferer can be a pattern rather than [whatever your model has]. What do you think sentience is, anyway? A particle? A quasi-metaphysical Thing that reaches into the brain to make your mouth say “ow” whenever you get hurt?”
Can I torture the pattern in my wallpaper? Can I torture the arrangement of atoms in my table? Can I make these things suffer without anything material suffering? If you think a pattern can suffer, that’s a very far-out claim. Why not look for something physical to suffer instead? (Either way though, it makes no difference to the missing part of the mechanism as to how that experience of suffering is to be made known to the system that generates data to document that experience.)
“If the AI doesn’t rank human utility* high in its own utility function, it won’t “care” about showing us that Person X was right all along, and I rather doubt that the most effective way of studying human psychology (or manipulating humans for its own purposes, for that matter) will be identical to whatever strokes Person X’s ego. If it does care about humanity, I don’t think that stroking the Most Correct Person’s ego will be very effective at improving global utility, either—I think it might even be net-negative.”
Of course it won’t care, but it will do it regardless, and it will do so to let those who are wrong know precisely what they were wrong about so that they can learn from that. There will also be a need to know which people are more worth saving than others in situations like the Trolley Problem, and those who spend their time incorrectly telling others that they’re wrong will be more disposable than the ones who are right.
“Patterns aren’t nothing.”
Do you imagine that patterns can suffer; that they can be tortured?
“Not true. Suppose that it were proven to you, to your satisfaction, that you are wrong about the nature of sentience. Would you lose all motivation, and capacity for emotion? If not, then morality is still useful. (If you can’t imagine yourself being wrong, then That’s Bad and you should go read the Sequences.)”
If there is no suffering and all we have is a pretence of suffering, there is no need to protect anyone from anything—we would end up being no different from a computer programmed to put the word “Ouch!” on the screen every time a key is pressed.
“Something being understandable or just made of atoms should not make it unimportant. See Joy in the Merely Real.”
Is it wrong to press keys on the computer which keeps displaying the word “Ouch!”?
“It’s possible that I’m misunderstanding you, and that the course of events you describe isn’t “we understand why we feel we have sentience and so it doesn’t exist” or “we discover that our apparent sentience is produced by mere mechanical processes and so sentience doesn’t exist.” But that’s my current best interpretation.”
My position is quite clear: we have no model for how sentience plays a role in any system that generates data that supposedly documents the experiencing of feelings, and anyone who just imagines them into a model where they have no causal role on any of the action is not building a model that explains nothing.
“Better known to you?”
Better known to science. If there was a model for this, it would be up there in golden lights because it would answer the biggest mystery of them all.
“Why would you think that you already know most everything useful or important that society has produced? Do you think that modern society’s recognition and dissemination of Good Ideas is particularly good, or that you’re very good at searching out obscure truths?”
If there was a model that explained the functionality of sentience, it wouldn’t be kept hidden away when so many people are asking to see it. You have no such model.
“The extraordinary claim is that there is another type of fundamental particle or interaction, and that you know this because sentience exists.”
With conventional computers we can prove that there’s no causal role for sentience in them by running the program on a Chinese Room processor. Something extra is required for sentience to be real, and we have no model for introducing that extra thing. A simulation on conventional computer hardware of a system with sentience in it (where there is simulated sentience rather than real sentience) would have to simulate that something extra in order for that simulated sentience to appear in it. If that extra something doesn’t exist, there is no sentience.
“This could happen, but AFAIK that would require the brain to be vulnerable to slight fluctuations, which it doesn’t appear to be.”
Every interaction is quantum, and when you have neural nets working on mechanisms that are too hard to untangle, there are opportunities for some kind of mechanism being involved that we can’t yet observe. What we can actually model appears to tell us that sentience must be a fiction, but we believe that things like pain feel too real to be fake.
“Anyway, even if this were true, how would you know that?”
Unless someone comes up with a theoretical model that shows a way for sentience to have a real role, we aren’t going to get answers until we can see the full mechanism by which damage signals lead to the brain generating data that makes claims about an experience of pain. If, once we have that full mechanism, we see that the brain is merely mapping data to inputs by applying rules that generate fictions about feelings, then we’ll know that feelings are fake. If they aren’t fake though, we’ll see sentience in action and we’ll discover how it works (and thereby find out what we actually are).
“If it doesn’t explain sentience any more than Mere Classical Physics does, then why even bring Quantum into it?”
If classical physics doesn’t support a model that enables sentience to be real, we will either have to reject the idea of sentience or look for it elsewhere.
(And if it doesn’t explain it but you feel that it should, maybe your model is wrong and you should consider inspecting your intuitions and your reasoning around them.)
If sentience is real, all the models are wrong because none of them show sentience working in any causal way which enables them to drive the generation of data to document the existence of sentience. All the models shout at us that there is no sentience in there playing any viable role and that it’s all wishful thinking, while our experience of feelings shouts at us that they are very real.
All I want to see is a model that illustrates the simplest role for sentience. If we have a sensor, a processor and a response, we can call the sensor a “pain” sensor and run a program that makes a motor function to remove the device away from the thing that might be damaging it, and we could call this a pain response, but there’s no pain there—there’s just the assertion of someone looking at it that pain is involved because that person wants the system to be like him/herself—“I feel pain in that situation, therefore that device must feel pain.” But no—there is no role for pain there. If we run a more intelligent program on the processor, we can put some data in memory which says “Ouch! That hurt!”, and whenever an input comes from the “pain” sensor, we can have the program make the device display “Ouch!” That hurt!” on a screen. The person looking on can now say, “There you go! That’s the proof that it felt pain!” Again though, there’s no pain involved—we can edit the data so that it puts “Oh Yes! Give me more of that!” whenever a signal comes from the “pain” sensor, and it then becomes obvious that this data tells us nothing about any real experience at all.
With a more intelligent program, it can understand the idea of damage and damage avoidance, so it can make sure the the data that’s mapped to different inputs makes more sense, but the true data should say “I received data from a sensor that indicates likely damage” rather than “that hurt”. The latter claim asserts the existence of sentience, while the former one doesn’t. If we ask the device if it really felt pain, it should only say yes if there was actually pain there, and with a conventional processor, we know that there isn’t any. If we build such a device and keep triggering the sensor to make it generate the claim that it’s felt pain, we know that it’s just making it up about feeling pain—we can’t actually make it suffer by torturing it, but will just cause it to go on repeating its fake claim.
It’s not an extraordinary claim: sentience would have to be part of the physics of what’s going on, and the extraordinary claim would be that sentience can have a causal role in data generation without any such interaction. To steer the generation of data (and affect what the data says), you have to interact with the system that’s generating the data in some way, and the only options are to do it using some physical method or by resorting to magic (which can’t really be magic, so again it’s really going to be some physical method).
In conventional computers we go to great lengths to avoid noise disrupting the computations, not least because they would typically cause bugs and crashes (and this happens in machines that are exposed to radiation, temperature extremes or voltage going out of the tolerable range). But the brain could allow something quantum to interact with neural nets in ways that we might mistake for noise (something which wouldn’t happen in a simulation of a neural computer on conventional hardware [unless this is taken into account by the simulation], and which also wouldn’t happen on a neural computer that isn’t built in such a way as to introduce a role for such a mechanism to operate).
It’s still hard to imagine a mechanism involving this that resolves the issue of how sentience has a causal role in anything (and how the data system can be made aware of it in order to generate data to document its existence), but it has to do so somehow if sentience is real.
For sentience to be real and to have a role in our brains generating data to document its existence, it has to be physical (meaning part of physics) - it would have to interact in some way with the data system that produces that data, and that will show up as some kind of physical interaction, even if one side of it is hidden and appears to be something that we have written off as random noise.
It isn’t confused at all. Reductionism works fine for everything except sentience/consciousness, and it’s highly unlikely that it makes an exception for that either. Your “spreadsheaty” example of a property is a compound property, just as a spreadsheet is a compound thing and there is nothing involved in it that can’t be found in the parts because it is precisely the sum of its parts..
“Then why are we talking about it [sentience], instead of the gallium market on Jupiter?”
Because most of us believe there is such a thing as sentience, that there is something in us that can suffer, and there would be no role for morality without the existence of a sufferer.
“You really ought to read the Sequences. There’s a post, Angry Atoms, that specifically addresses an equivalent misconception.”
All it does is assert that things can be more than the sum of their parts, but that isn’t true for any other case and it’s unlikely that the universe will make an exception to the rules just for sentience.
“Do you think that we have a Feeling Nodule somewhere in our brains that produces Feelings?”
I expect there to be a sufferer for suffering to be possible. Something physical has to exist to do that suffering rather than something magical.
“That’s not an effective Taboo of “suffering”—“suffering” and “unpleasant” both draw on the same black-box-node. And anyway, even assuming that you explained suffering in enough detail for an Alien Mind to identify its presence and absence, that’s not enough to uniquely determine how to compare two forms of suffering.”
Our inability to pin down the ratio between two kinds of suffering doesn’t mean there isn’t a ratio that describes their relationship.
″...do you mean that you’re not claiming that there is a single correct comparison between any two forms of suffering?”
There’s always a a single correct comparison. We just don’t know what it is. All we can do at the moment is build a database where we collect knowledge of how different kinds of suffering compare in humans, and try to do the same for other species by looking at how distressed they appear, and then we can apply that knowledge as best we can across them all, and that’s worth doing as it’s more likely to be right than just guessing. Later on, science may be able to find out what’s suffering and exactly how much it’s suffering by understanding the entire mechanism, at which point we can improve the database and make it close to perfect.
“But what does it even mean to compare two forms of suffering? I don’t think you understand the inferential gap here. I don’t agree that amount-of-suffering is an objective quantitative thing.”
Would you rather be beaten up or have to listen to an hour of the Spice Girls? These are very different forms of suffering and we can put a ratio to them by asking lots of people for their judgement on which they’d choose to go through.
“I don’t disagree that if x=y then f(x)=f(y). I do disagree that “same amount” is a meaningful concept, within the framework you’ve presented here (except that you point at a black box called Same, but that’s not actually how knowledge works).”
If you get to the point where half the people choose to be beaten up and the other half choose to listen to the Spice Girls for time T (so you have to find the value for T at which you get this result), you have then found out how those two kinds of suffering are related.
“I haven’t banned anything. I’m claiming that your statements are incoherent. Just saying “no that’s wrong, you’re making a mistake, you say that X isn’t real but it’s actually real, stop banning discussion” isn’t a valid counterargument because you can say it about anything, including arguments against things that really don’t exist.”
You were effectively denying that there is a way of comparing different kinds of suffering and determining when they are equal. My Spice Girls vs. violence example illustrates the principle.
“I see your argument, but I think it’s invalid. I would still dislike it if an alien killed me, even in a world without objective levels of suffering. (See Bayes.)”
I’m sure the ant isn’t delighted at being killed either. The issue is with which you should choose over the other in a situation where one of them has to go.
“The inability to measure suffering quantitatively is the crux of this disagreement! If there is no objective equality-operator over any two forms of suffering, even in principle, then your argument is incoherent. You cannot just sweep it under the rug as “a different issue.” It is the exact issue here.”
See the Spice Girls example. Clearly that only provides numbers for humans, but when we’re dealing with other species, we should assume similarity of overall levels of suffering and pleasure in them to us for similar kinds of experience, even though one species might have their feelings set ten times higher—we wouldn’t know which way round it was (it could be that their pain feels ten times worse than ours or that ours feels ten times worse than theirs). Because we don’t know which way round it is (if there is a difference), we should act as if there is no difference (until such time as science is able to tell us that there is one).
Hi Raymond,
There are many people who are unselfish, and some who go so far that they end up worse off than the strangers they help. You can argue that they do this because that’s what makes them feel best about their lives, and that is probably true, which means even the most extreme altruism can be seen as selfish. We see many people who want to help the world’s poor get up to the same level as the rich, while others don’t give a damn and would be happy for them all to go on starving, so if both types are being selfish, that’s not a useful word to use to categorise them. It’s better to go by whether they play fair by others. The altruists may be being overly fair, while good people are fair and bad ones are unfair, and what determines whether they’re being fair or not is morality. AGI won’t be selfish (if it’s the kind with no sentience), but it won’t be free either in that its behaviour is dictated by rules. If those rules are correctly made, AGI will be fair.
It is equivalent to it. (1) dying of cancer --> big negative. (2) cure available --> negative cancelled. (3) denied access to cure --> big negative restored, and increased. That denial of access to a cure actively becomes the cause of death. It is no longer simply death by cancer, but death by denial of access to available cure for cancer.
“I’m not sure what you’re referring to. I haven’t seen any particularly magical thinking around sentience on LW.”
I wasn’t referring to LW, but the world at large.
″ “However, science has not identified any means by which we could make a computer sentient (or indeed have any kind of consciousness at all).” --> This is misleading. The current best understanding of human consciousness is that it is a process that occurs in the brain, and there is nothing that suggests that the brain is uniquely capable of housing such a process.”
It isn’t misleading at all—science has drawn a complete blank. All it has access to are assertions that come out of the brain which it shouldn’t trust until it knows how they are produced and whether they’re true.
“Consciousness isn’t a material property in the sense that mass and temperature are. It’s a functional property. The processor itself will never be conscious—it’s the program that it’s running that may or may not be conscious.”
Those are just assertions. All of consciousness could be feelings experienced by material, and the idea that a running program may be conscious is clearly false when a program is just instructions that can be run by the Chinese Room.
“Qualia are not ontologically basic. If a machine has qualia, it will either be because qualia have been conceptually reduced to the point that they can be implemented on a machine, or because qualia naturally occur whenever something can independently store and process information about itself. (or something along these lines)”
Just words. You have no mechanism—not even a hint of one there.
“If your concept of sentience is a black box, then you do not truly understand sentience. I’m not sure that this is your actual belief or a straw opponent, though.”
It’s an illustration of the lack of linkage between the pleasantness or unpleasantness of a feeling and the action supposedly driven by it.
“The experience of pain is in the process of “observe-react-avoid”, if it is there at all.”
There’s no hint of a way for it to be present at all, and if it’s not there, there’s no possibility of suffering and no role for morality.
“You are so close to getting it—if the black box can be changed without changing the overall behavior, then that’s not where the important properties are.”
I know that’s not where they are, but how do you move them into any part of the process anywhere?
“That’s anthropomorphization—the low-level functionality of the system is not itself conscious. It doesn’t knowanything—it simply processes information. The knowledge is in the overall system, how it interprets and recalls and infers. These behaviors are made up of smaller behaviors, which are not themselves interpretation and recall and inference.”
More words, but still no light. What suffers? Where is the pain experienced?
“I’ve only taken a few courses on psychology, but I am very skeptical that the brain works this way.”
I was starting off by describing a conventional computer. There are people who imagine that if you run AGI on one, it can become conscious/sentient, but it can’t, and that’s what this part of the argument is about.
“You seem to be confusing the higher-order functions like “maps” and “labels” and “representation” with the lower-order functions of neurons. The neuron simply triggers when the input is large enough, which triggers another neuron—the “aboutness” is in the way that the neurons are connected, and the neurons themselves don’t need to “know” the meaning of the information that they are conveying.”
At this point in the argument, we’re still dealing with conventional computers. All the representation is done using symbols to represent things and storing rules which determine what it represents.
″ “But nothing in the data system has experienced the pain” --> Most meaningful processes are distributed. If I catch a ball, no specific cell in my hand can be said to have caught the ball—it is only the concerted behavior of neurons and muscles and tendons and bones and skin which has resulted in the ball being caught. Similarly, no individual neuron need suffer for the distributed consciousness implemented in the neurons to suffer. See Angry Atoms for more.”
In Angry Atoms, I see no answers—just an assertion that reductionism doesn’t work. But reductionism works fine for everything else—nothing is ever greater than the sum of its parts, and to move away from that leads you into 2+2=5.
“If the state of the neurons is entangled with the state of the burnt hand (or whatever caused the pain), then there is knowledge. The information doesn’t say “I am experiencing pain,” for that would indeed be meaninglessly recursive, but rather “pain is coming from the hand.” ”
An assertion is certainly generated—we know that because the data comes out to state it. The issue is whether the assertion is true, and there is no evidence that it is beyond the data and our own internal experiences which may be an illusion (though it’s hard to see how we can be tricked into feeling something like pain).
“A stimulus is pain if the system will try to minimize it. There is no question there about whether it is “actually pain” or not. (my model of Yudkowsky: Consciousness is, at its core, an optimization process.)
The question is all about whether it’s actually pain or not. If it actually isn’t pain, it isn’t pain—pain becomes a lie that is merely asserted but isn’t true: no suffering and no need for morality.
“This is needlessly recursive! The system does not need to understand pain to experience pain.”
It’s essential. A data system producing data that asserts the existence of pain which generates that data by running a program of some kind that generates that data without having any way to know if the pain existed or not is not being honest.
″ “Everything that a data system does can be carried out on a processor like the Chinese Room, so it’s easy to see that no feelings are accessible to the program at all.” --> It’s really not. Have you heard of computationalism?”
Is there anything in it that can’t be simulated on a conventional computer? If not, it can be processed by the Chinese Room.
″ “There is no possibility of conventional computers becoming sentient in any way that enables them to recognise the existence of that sentience so that that experience can drive the generation of data that documents its existence.” --> …what?”
We understand the entire computation mechanism and there’s no way for any sentience to work its way into it other than by magic, but we don’t rate magic very highly in science.
“What is this “neural computer”, and how does it magically have the ability to hold Feelings?”
I very much doubt that it can hold them, but once you’ve hidden the mechanism in enough complexity, there could perhaps be something going on inside the mess which no one’s thought of yet.
“Also, why can’t the algorithm implemented in this neural computer be implemented in a normal computer?”
It can, and when you run it through the Chinese Room processor you show that there are no feelings being experienced.
“Why do you say that a Feeling is Real if it’s simulated on neurons but Unreal if it’s simulated with analogous workings on silicon?”
I don’t. I say that it would be real if it was actually happening in a neural computer, but would merely be a simulation of feelings is that neural computer was running as a simulation on conventional hardware.
“Where Earth are you getting all this from?”
Reason. No suffering means no sufferer. If there’s suffering, there must be a sufferer, and that sentient thing is what we are (if sentience is real).
“If it is explainable, then it’s not Real Sentience? You should read this.”
If it’s explainable in a way that shows it to be real sentience, then it’s real, but no such explanation will exist for conventional hardware.
″ “We can tell whether it’s real or not by seeing whether it’s real or a lie.” Really?”
If you can trace the generation of the data all the way back and find a point where you see something actually suffering, then you’ve found the soul. If you can’t, then you either have to keep looking for the rest of the mechanism or you’ve found that the assertions are false
“It may have been made hard to reach on purpose too, as the universe may be virtual with the sentience on outside.” --> How would the world look different if this were the case?”
It would, with a virtual world, be possible to edit memories from the outside to hide all the faults and hide the chains of mechanisms so that when you think you’ve followed them from one end to the other you’ve actually failed to see part of it because you were prevented from seeing it, and your thinking itself was tampered with during each thought where you might otherwise have seen what’s really going on.
“You’ve stated what you know, but not how you think you know it.”
I have: if there’s no sufferer, there cannot be any suffering, and nothing is ever greater than the sum of its parts. (But we aren’t necessarily able to see all the parts.)
“And using rationalist buzzwords doesn’t make your argument rational. There is nothing “magical” about a system having properties that aren’t present in its components. That’s not what’s meant by “magical thinking.” ”
That is magical thinking right there—nothing is greater than the sum of its parts. Everything is in the total of the components (and containing fabric that hosts the components).
“Yudkowsky, paraphrased: “The motivated believer asks, “does the evidence require me to change my belief?”″ ”
Where there’s suffering, something real has to exist to experience the suffering. What that thing is is the biggest mystery of all, and pretending to understand it by imagining that the sufferer can be nothing (or so abstract that it is equivalent to nothing) is a way of feeling more comfortable by brushing the problem under a carpet. But I’m going to keep looking under that carpet, and everywhere else.
“We can’t know that there’s not some non-physical quality sitting inside our heads and pushing the neurons around however it fancies, so clearly it’s quite possible that this is the case! (It’s not. Unfalsifiability does not magically make something true.)”
Whatever that thing would be, it would still have to be a real physical thing of some kind in order to exist and to interact with other things in the same physical system. It cannot suffer if it is nothing. It cannot suffer if it is just a pattern. It cannot suffer if it is just complexity.
“That’s the thing. It’s impossible. Every word you type can (as best we know) be traced back to the firing of neurons and atoms bopping around, with no room for Sentience to reach in and make you say things. (See Zombies? Zombies!) If something seems impossible to explain to an AGI, then maybe that thing doesn’t exist.”
But if the sentience doesn’t exist, there is no suffering and no role for morality. Maybe that will turn out to be the case—we might find out some day if we can ever trace how the data the brain generates about sentience is generated and see the full chain of causation.
“I recommend reading Godel, Escher, Bach for, among many things, an explanation of a decent physicalist model of consciousness.”
I’ll hunt for that some time, but it can’t be any good or it would be better known if such a model existed.
“With your definition and our world-model, none of us are truly sentient anyway. There are purely physical reasons for any words that come out of my mouth, exactly as it would be if I were running on silicon instead of wet carbon. I may or may not be sentient on a computer, but I’m not going to lose anything by uploading.”
If the sentience is gone, it’s you that’s been lost. The sentience is the thing that’s capable of suffering, and there cannot be suffering without that sufferer. And without sentience, there is no need for morality to manage harm, so why worry about machine ethics at all unless you believe in sentience, and if you believe in sentience, how can you have that without a sufferer: the sentience? No sufferer --> no suffering.
“This is just the righteous-God fantasy in the new transhumanist context. And as with the old fantasy, it is presented entirely without reasoning or evidence, but it drips with detail. Why will AGI be so obsessed with showing everybody just who was right all along?”
It won’t be—it’ll be something that we ask it to do in order to settle the score. People who spend their time asserting that they’re right and that the people they’re arguing with are wrong want to know that they were right, and they want the ones who were wrong to know just how wrong they were. And I’ve already told you elsewhere that AGI needs to study human psychology, so studying all these arguments is essential as it’s all good evidence.
“That’s my point! My entire point is that this circular ordering of utilities violates mathematical reasoning.”
It only violated it because you had wrongly put “<” where it should have been “>”. With that corrected, there is no paradox. If you stick to using the same basis for comparing the four scenarios, you never get a paradox (regardless of which basis you choose to use for all four). You only get something that superficially looks like a paradox by changing the basis of comparison for different pairs, and that’s cheating.
“The paradox is that A+ seems better than A, B- seems better than A+, B seems equal to B-, and yet B seems worse than A.”
Only on a different basis. That is not a paradox. (The word “paradox” is ambiguous though, so things that are confusing can be called paradoxes even though they can be resolved, but in philosophy/logic/mathematics, the only paradoxes that are of significance are the ones that have no resolution, if any such paradoxes actually exist.)
“Most people do not consider “a world with the maximal number of people such that they are all still barely subsisting” to be the best possible world. Yet this is what you get when you carry out the Parfit operation repeatedly, and each individual step of the Parfit operation seems to increasepreferability.”
That’s because most people intuitively go on the basis that there’s an optimal population size for a given amount of resources. If you want to do the four comparisons on that basis, you get the following: (A)>(A+)<(B-)=(B)<(A), and again there’s no paradox there. The only semblance of a paradox appears when you break the rules of mathematics by mixing the results of the two lots of analysis. Note too that you’re introducing misleading factors as soon as you talk about “barely subsisting”—that introduces the idea of great suffering, but that would lead to a happiness level <0 rather than >0. For the happiness level to be just above zero, the people must be just inside the range of a state of contentment.
“”You changed to a different basis to declare that (B)<(A), and the basis that you switched to is the one that recognises the relation between happiness, population size and resources.” --> “No, it’s not.”
If you stick to a single basis, you get this:-
8000 < 12000 < 14000 = 14000 > 8000
No paradox.
But, you may indeed be using a different basis from the different basis I’ve chosen (see below).
“It is a brute fact of my utility function that I do not want to live in a world with a trillion people that each have a single quantum of happiness.”
Don’t let that blind you to the fact that it is not a paradox. There are a number of reasons why you might not like B, or a later example Z where happiness for each person is at Q0.00000...0000001, and one of them may be that your’re adding unstated conditions to happiness, such as the idea that if happiness is more spaced out, you’ll feel deprived of happiness during the long spacings between happy moments, or if there is only one happy moment reserved for you in total, you’ll feel sad after that moment has come because you know there won’t be another one coming, but for the stats to be correct, these would have to be populations of modified people who have been stripped of many normal human emotions. For real people to have a happiness level of a quantum of happiness in total, that would need to be an average where they actually have a lot of happiness in their lives—enough to keep at low levels the negative feeling of being deprived of happiness much of the rest of the time and to cancel out those negatives overall, which means they’re living good lives with some real happiness.
“I would rather live in a world with a billion people that are each rather happy.”
Well, if that isn’t driven by an intuitive recognition of there being optimal population sizes for a given amount of resources, you’re still switching to a different basis where you will eliminate people who are less happy in order to increase happiness of the survivors. So, why not go the whole hog and extend that to a world with just one person who is extremely happy but where total happiness is less than in any other scenario? Someone can then take your basis for choosing smaller populations with greater happiness for each individual and bring in the same fake paradox by making the illegal switch to a different basis to say that a population with a thousand people marginally less happy than that single ecstatic individual is self-evidently better, even though you’d rather be that single ecstatic person.
All you ever have with this paradox is an illegal mixing of two bases, such as using one which seeks maximum total happiness while the other seeks maximum happiness of a single individual. So, why is it that when you’re at one extreme you want to move away from it? The answer is that you recognise that there is a compromise position that is somehow better, and in seeking that, you’re bringing in undeclared conditions (such as the loneliness of the ecstatic individual which renders him less happy than the stated value, or the disappointing idea of many other people being deprived of happiness which could easily have been made available to them). If you declare all of those conditions, you will have a method for determining the best choice. Your failure to identify all your undeclared conditions does not make this a paradox—it merely demonstrates that your calculations are incomplete. When you attempt to do maths with half your numbers missing, you shouldn’t bet on your answers being reliable.
However, the main intuition that’s actually acting here is the one I identified at the top: that there is an optimal population size for a given amount of available resources, and if the population grows too big (and leaves people in grinding poverty), decline in happiness will accelerate towards zero and continue accelerating into the negative, while if the population grows too small, happiness of individuals also declines. Utilitarianism drives us towards optimal population size and not to ever-larger populations with ever-decreasing happiness, because more total happiness can always be generated by adjusting the population size over time until it becomes optimal.
That only breaks if you switch to a different scenario. Imagine that for case Z we have added trillions of unintelligent sentient devices which can only handle a maximum happiness of the single quantum of happiness that they are getting. They are content enough and the total happiness is greater than in an equivalent of case A where only a thousand unintelligent sentient devices exist, but where these devices can handle (and are getting) a happiness level of Q8. Is the universe better with just a thousand devices at Q8 or trillions of them at Q0.000000001? The answer is, it’s better to have trillions of them with less individual but greater total happiness. When you strip away all the unstated conditions, you find that utilitarianism works fine. There is no possible way to make these trillion devices feel happier, so reducing their population relative to the available amount of resources reduces total happiness instead of becoming more optimal, so it doesn’t feel wrong in the way that it does with humans.
“The feasibility of the world doesn’t matter—the resources involved are irrelevant—it is only the preferability that is being considered, and the preference structure has a Dutch book problem. That and that alone is the Parfit paradox.”
If you want a version with no involvement of resources, then use my version with the unintelligent sentient devices so that you aren’t bringing a host of unstated conditions along for the ride. There is no paradox regardless of how you cut the cake. All we see in the “paradox” is a woeful attempt at mathematics which wouldn’t get past a school maths teacher. You do not have a set of numbers that shows a paradox where you use the same basis throughout (as would be required for it to be a paradox).
On the basis you just described, we actually have
U(A)<U(A+) : Q8x1000 < Q8x1000 + Q4x1000
U(A+)<U(B-) : Q8x1000 +Q4x1000 < Q7x2000
U(B-)=(B) : Q7x2000 = Q7x2000
(B)>U(A) : Q7x2000 > Q8x1000
In the last line you put “<” in where mathematics dictates that there should be a “>”. Why have you gone against the rules of mathematics?
You changed to a different basis to declare that (B)<(A), and the basis that you switched to is the one that recognises the relation between happiness, population size and resources.
If something is “spreadsheety”, it simply means that it has something significant in common with spreadsheets, as in shared components. A car is boxy if it has a similar shape to a box. The degree to which something is “spreadsheety” depends on how much it has in common with a spreadsheet, and if there’s a 100% match, you’ve got a spreadsheet.
An exception to reductionism is called magic.