Roko’s reply to me strongly suggested that he interpreted my message as requesting deletion, and that I was the cause of him deleting it. I doubt anyone at SIAI would have explicitly requested deletion.
I’m wondering whether you, Nick, have learned anything from this experience—something perhaps about how attempting to hide something is almost always counterproductive?
Of course, Roko contributed here by deleting the message, you didn’t create this mess by yourself. But you sure have helped. :)
Well, look, I deleted it of my own accord, but only after being prompted that it was a bad thing to have posted. Can we just drop this? It makes me look like even more of a troublemaker than I already look like, and all I really want to do is finish the efficient charity competition then get on with life outside teh intenetz.
Will you at least publicly state that you precommit, on behalf of CEV, to not apply negative incentives in this case? (Roko, Jul 24, 2010 1:37 PM)
This is very important. If the SIAI is the organisation to solve the friendly AI problem and implement CEV then it should be subject to public examination, especially if they ask for money.
The current evidence that anyone anywhere can implement CEV is two papers in six years that talk about it a bit. There appears to have been nothing else from SIAI and no-one else in philosophy appears interested.
If that’s all there is for CEV in six years, and AI is on the order of thirty years away, then (approximately) we’re dead.
This is rather disappointing, as if CEV is possible then a non-artificial general intelligence should be able to implement it, at least partially. And we have those. The reason for CEV is (as I understand it) the danger of the AI going FOOM before it cares about humans. However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
Addendum: I see others have been asking “but what do you actually mean?” for a couple of years now.
The current evidence that anyone anywhere can implement CEV is two papers in six years that talk about it a bit. There appears to have been nothing else from SIAI and no-one else in philosophy appears interested.
If that’s all there is for CEV in six years, and AI is on the order of thirty years away, then (approximately) we’re dead.
This strikes me as a demand for particular proof. SIAI is small (and was much smaller until the last year or two), the set of people engaged in FAI research is smaller, Eliezer has chosen to focus on writing about rationality over research for nearly four years, and FAI is a huge problem, in which any specific subproblem should be expected to be underdeveloped at this early stage. And while I and others expect work to speed up in the near future with Eliezer’s attention and better organization, yes, we probably are dead.
The reason for CEV is (as I understand it) the danger of the AI going FOOM before it cares about humans.
Somewhat nitpickingly, this is a reason for FAI in general. CEV is attractive mostly for moving as much work from the designers to the FAI as possible, reducing the potential for uncorrectable error, and being fairer than letting the designers lay out an object-level goal system.
This is rather disappointing, as if CEV is possible then a non-artificial general intelligence should be able to implement it, at least partially.… However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
This sounds interesting; do you think you could expand?
It wasn’t intended to be—more incredulity. I thought this was a really important piece of the puzzle, so expected there’d be something at all by now. I appreciate your point: that this is a ridiculously huge problem and SIAI is ridiculously small.
However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
This sounds interesting; do you think you could expand?
I meant that, as I understand it, CEV is what is fed to the seed AI. Or the AI does the work to ascertain the CEV. It requires an intelligence to ascertain the CEV, but I’d think the ascertaining process would be reasonably set out once we had an intelligence on hand, artificial or no. Or the process to get to the ascertaining process.
I thought we needed the CEV before the AI goes FOOM, because it’s too late after. That implies it doesn’t take a superintelligence to work it out.
Thus: CEV would have to be a process that mere human-level intelligences could apply. That would be a useful process to have, and doesn’t require first creating an AI.
I must point out that my statements on the subject are based in curiosity, ignorance and extrapolation from what little I do know, and I’m asking (probably annoyingly) for more to work with.
“CEV” can (unfortunately) refer to either CEV the process of determining what humans would want if we knew more etc., or the volition of humanity output by running that process. It sounds to me like you’re conflating these. The process is part of the seed AI and is needed before it goes FOOM, but the output naturally is neither, and there’s no guarantee or demand that the process be capable of being executed by humans.
I have recieved assurances that SIAI will go to significant efforts not to do nasty things, and I believe them. Private assurances given sincerely are, in my opinion, the best we can hope for, and better than we are likely to get from any other entity involved in this.
Besides, I think that XiXiDu, et al are complaining about the difference between cotton and silk, when what is actually likely to happen is more like a big kick in the teeth from reality. SIAI is imperfect. Yes. Well done. Nothing is perfect. At least cut them a bit of slack.
I have recieved assurances that SIAI will go to significant efforts not to do nasty things, and I believe them. Private assurances given sincerely are the best we can hope for, and better than we are likely to get from any other entity involved in this.
What?!? Open source code—under a permissive license—is the traditional way to signal that you are not going to run off into the sunset with the fruits of a programming effort. Private assurances are usually worth diddly-squat by comparison.
I think that you don’t realize just how bad the situation is. You want that silken sheet. Rude awakening methinks. Also open-source not neccessarily good for FAI in any case.
I think that you don’t realize just how bad the situation is.
I don’t think that you realize how bad it is. I’d rather have the universe being paperclipped than supporting the SIAI if that means that I might be tortured for the rest of infinity!
To the best of my knowledge, SIAI has not planned to do anything, under any circumstances, which would increase the probability of you or anyone else being tortured for the rest of infinity.
Supporting SIAI should not, to the best of my knowledge, increase the probability of you or anyone else being tortured for the rest of infinity.
But imagine there was a person a level above yours that went to create some safeguards for an AGI. That person would tell you that you can be sure that the safeguards s/he plans to implement will benefit everyone. Are you just going to believe that? Wouldn’t you be worried and demand that their project is being supervised?
You are in a really powerful position because you are working for an organisation that might influence the future of the universe. Is it really weird to be skeptical and ask for reassurance of their objectives?
Logical rudeness is the error of rejecting an argument for reasons other than disagreement with it. Does your “I don’t think so” mean that you in fact believe that SIAI (possibly) plans to increase the probability of you or someone else being tortured for the rest of eternity? If not, what does this statement mean?
I removed that sentence. I meant that I didn’t believe that the SIAI plans to harm someone deliberately. Although I believe that harm could be a side-effect and that they would rather harm a few beings than allowing some Paperclip maximizer to take over.
You can call me a hypocrite because I’m in favor of animal experiments to support my own survival. But I’m not sure if I’d like to have someone leading an AI project who thinks like me. Take that sentence to reflect my inner conflict. I see why one would favor torture over dust specks but I don’t like such decisions. I’d rather have the universe to end now, or having everyone turned into paperclips, than having to torture beings (especially if I am the being).
I feel uncomfortable that I don’t know what will happen because there is a policy of censorship being favored when it comes to certain thought experiments. I believe that even given negative consequences, transparency is the way to go here. If the stakes are this high, people who believe will do anything to get what they want. That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
Better yet, you could use a kind of doublethink—and then even actually mean it. Here is W. D. Hamilton on that topic:
A world where everyone else has been persuaded to be altruistic is a good one to live in from the point of view of pursuing our own selfish ends. This hypocracy is even more convincing if we don’t admit it even in our thoughts—if only on our death beds, so to speak, we change our wills back to favour the carriers of our own genes.
Discriminating Nepotism—as reprinted in: Narrow Roads of Gene Land, Volume 2 Evolution of Sex, p.356.
That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
For 30 years I have been wondering, what indication of its existence might we expect from a true AI? Certainly not any explicit revelation, which might spark a movement to pull the plug. Anomalous accumulation or creation of wealth might be a sign, or an unquenchable thirst for raw information, storage space, and processing cycles, or a concerted attempt to secure an uninterrupted, autonomous power supply. But the real sign, I suspect, would be a circle of cheerful, contented, intellectually and physically well-nourished people surrounding the AI.
I think many people would like to be in that group—if they can find a way to arrange it.
Unless AI was given that outcome (cheerful, contented people etc) as a terminal goal, or that circle of people was the best possible route to some other terminal goal, both of which are staggeringly unlikely, Dyson suspects wrongly.
If you think he suspects rightly, I would really like to see a justification. Keep in mind that AGIs are currently not being built using multi-agent environment evolutionary methods, so any kind of ‘social cooperation’ mechanism will not arise.
Machine intelligence programmers seem likely to construct their machines so as to help them satisfy their preferences—which in turn is likely to make them satisfied. I am not sure what you are talking about—but surely this kind of thing is already happening all the time—with Sergey Brin, James Harris Simons—and so on.
That doesn’t really strike me as a stunning insight, though. I have a feeling that I could find many people who would like to be in almost any group of “cheerful, contented, intellectually and physically well-nourished people.”
This all depends on what the AI wants. Without some idea of its utility function, can we really speculate? And if we speculate, we should note those assumptions. People often think of an AI as being essentially human-like in its values, which is problematic.
It’s a fair description of today’s more successful IT companies. The most obvious extrapolation for the immediate future involves more of the same—but with even greater wealth and power inequalities. However, I would certainly also council caution if extrapolating this out more than 20 years or so.
Currently, there are no entities in physical existence which, to my knowledge, have the ability to torture anyone for the rest of eternity.
You intend to build an entity which would have that ability (or if not for infinity, for a googolplex of subjective years).
You intend to give it a morality based on the massed wishes of humanity—and I have noticed that other people don’t always have my best interests at heart. It is possible—though unlikely—that I might so irritate the rest of humanity that they wish me to be tortured forever.
Therefore, you are, by your own statements, raising the risk of my infinite torture from zero to a tiny non-zero probability. It may well be that you are also raising my expected reward enough for that to be more than counterbalanced, but that’s not what you’re saying—any support for SIAI will, unless I’m completely misunderstanding, raise the probability of infinite torture for some individuals.
That just pushes the problem along a step. IF the Last Judge can’t be mistaken about the results of the AI running AND the Last Judge is willing to sacrifice the utility of the mass of humanity (including hirself) to protect one or more people from being tortured, then it’s safe. That’s very far from saying there’s a zero probability.
IF … the Last Judge is willing to sacrifice the utility of the mass of humanity (including hirself) to protect one or more people from being tortured, then it’s safe.
If the Last Judge peeks at the output and finds that it’s going to decide to torture people, that doesn’t imply abandoning FAI, it just requires fixing the bug and trying again.
Just because AGIs have capability to inflict infinite torture, doesn’t mean they have a motive. Also, status quo (with regard to SIAI’s activity) doesn’t involve nothing new happening.
I explained that he is planning to supply one with a possible motive (namely that the CEV of humanity might hate me or people like me). It is precisely because of this that the problem arises. A paperclipper, or any other AGI whose utility function had nothing to do with humanity’s wishes, would have far less motive to do this—it might kill me, but it really would have no motive to torture me.
If someone solicits for you to “trust in me”, alarm bells should start ringing immediately. If you really think that is “the best we can hope for”, then perhaps revisit that.
Gollum gave his private assurances to Frodo—and we all know how that turned out.
Well I’m convinced. Frodo should definitely have worked out a way to clone the ring and made sure the information was available to all of Middle Earth. You can never have too many potential Ring-Wraiths.
I’d also like to see His Dark Materials with rationalist!Lyra. The girl had an alethiometer. She should have kicked way more ass than she did as soon as she realized what she had.
Open source AGI is not a good thing. In fact, it would be a disastrously bad thing. Giving people the source code doesn’t just let them inspect it for errors, it also lets them launch it themselves. If you get an AGI close to ready for launch, then sharing its source code means that instead of having one party to decide whether there are enough safety measures ready to launch, you have many parties individually deciding whether to launch it themselves, possibly modifying its utility function to suit their own whim, and the hastiest party’s AGI wins.
Ideally, you’d want to let people study the code, but only trustworthy people, and in a controlled environment where they can’t take the source code with them. But even that is risky, since revealing that you have an AGI makes you a target for espionage and attack by parties who shouldn’t be trusted with humanity’s future.
Actually it reduces the chance of any party drawing massively ahead of the rest. It acts as an equalising force, by power-sharing. Since one of the main things we want to avoid is a disreputable organisation using machine intelligence to gain an advantage—and sustaining it over a long period of time. Using open-source software helps to defend against that possibility.
Machine intelligence will be a race—but it will be a race, whether participants share code or not.
Having said all that, machine intelligence protected by patents with secret source code on a server somewhere does seem like a reasonably probable outcome.
Using open-source software helps to defend against that possibility.
Only if (a) there is no point at which AGIs “foom”, (b) source code sharing is well enough enforced on everyone that no bad organizations combine open source with refinements that they keep secret for an advantage, (c) competing AIs form a stable power equilibrium at all points along their advancement, and (d) it is impossible to trade off goal system stability for optimization power.
I estimate probabilities of 0.4, 0.2, 0.05, and 0.5 for these hypotheses, respectively.
I disagree with most of that analysis. I assume machine intelligence will catalyse its own creation. I fully expect that some organisations will stick with secret source code. How could the probability of that possibly be as low as 0.8!?!
I figure that use of open source software is more likely to lead to a more even balance of power—and less likely to lead to a corrupt organisation in charge of the planet’s most advanced machine intelligence efforts. That assessment is mostly based on the software industry to date—where many of the worst abuses appear to me to have occurred at the hands of proprietary software vendors.
If you have an unethical open source project, people can just fork it, and make an ethical version. With a closed source project, people don’t have that option—they often have to go with whatever they are given by those in charge of the project.
Nor am I assuming that no team will ever win. If there is to be a winner, we want the best possible lead up. The “trust us” model is not it—not by a long shot.
I figure that use of open source software is more likely to lead to a more even balance of power—and less likely to lead to a corrupt organisation in charge of the planet’s most advanced machine intelligence efforts. That assessment is mostly based on the software industry to date—where many of the worst abuses appear to me to have occurred at the hands of proprietary software vendors.
If you have an unethical open source project, people can just fork it, and make an ethical version. With a closed source project, people don’t have that option—they often have to go with whatever they are given by those in charge of the project.
There are two problems with this reasoning. First, you have the causality backwards: makers of open-source software are less abusive than makers of closed-source software not because open-source is such a good safeguard, but because the sorts of organizations that would be abusive don’t open source in the first place.
And second, if there is an unethical AI running somewhere, then forking the code will not save humanity. Forking is a defense against not having good software to use yourself; it is not a defense against other people running software that does bad things to you.
you have the causality backwards: makers of open-source software are less abusive than makers of closed-source software not because open-source is such a good safeguard, but because the sorts of organizations that would be abusive don’t open source in the first place.
Really? I just provided an example of a mechanism that helps keep open source software projects ethical—the fact that if the manufacturers attempt to exploit their customers it is much easier for the customers to switch to a more ethical fork—because creating such a fork no longer violates copyright law. Though you said you were pointing out problems with my reasoning, you didn’t actually point out any problems with that reasoning.
We saw an example of this kind of thing very recently—with LibreOffice. The developers got afraid that their adopted custodian, Oracle, was going to screw the customers of their project—so, to protect their customers and themselves, they forked it—and went their own way.
if there is an unethical AI running somewhere, then forking the code will not save humanity. Forking is a defense against not having good software to use yourself; it is not a defense against other people running software that does bad things to you.
If other people are running software that does bad things to you then running good quality software yourself most certainly is a kind of defense. It means you are better able to construct defenses, better able to anticipate their attacks—and so on. Better brains makes you more powerful.
Compare with the closed-source alternative: If other people are running software that does bad things to you—and you have no way to run such software yourself—since it is on their server and running secret source that is also protected by copyright law—you are probably pretty screwed.
It makes me look like even more of a troublemaker...
How so? I’ve just reread some of your comments on your now deleted post. It looks like you honestly tried to get the SIAI to put safeguards into CEV. Given that the idea spread to many people by now, don’t you think it would be acceptably to discuss the matter before one or more people take it serious or even consider to implement it deliberately?
Ok by me. It is pretty obvious by this point that there is no evil conspiracy involved here. But I think the lesson remains, I you delete something, even if it is just because you regret posting it, you create more confusion than you remove.
I think the question you should be asking is less about evil conspiracies, and more about what kind of organization SIAI is—what would they tell you about, and what would they lie to you about.
If the forbidden topic would be made public (and people would believe it), it would result in a steep rise of donations towards the SIAI. That alone is enough to conclude that the SIAI is not trying to hold back something that would discredit it as an organisation concerned with charitable objectives. The censoring of the information was in accordance with their goal of trying to prevent unfriendly artificial intelligence. Making the subject matter public did already harm some people and could harm people in future.
But the forbidden topic is already public. All the effects that would follow from it being public would already follow. THE HORSE HAS BOLTED. It’s entirely unclear to me what pretending it hasn’t does for the problem or the credibility of the SIAI.
Of course, if Clippy were clever he would then offer to sell SIAI a commitment to never release the UFAI in exchange for a commitment to produce a fixed number of paperclips per year, in perpetuity.
Admittedly, his mastery of human signaling probably isn’t nuanced enough to prevent that from sounding like blackmail.
If the forbidden topic would be made public (and people would believe it), it would result in a steep rise of donations towards the SIAI.
I really don’t see how that follows. Will more of the public take it seriously? As I have noted, so far the reaction from people outside SIAI/LW has been “They did WHAT? Are they IDIOTS?”
The censoring of the information was in accordance with their goal of trying to prevent unfriendly artificial intelligence.
That doesn’t make it not stupid or not counterproductive. Sincere stupidity is not less stupid than insincere stupidity. Indeed, sincere stupidity is more problematic in my experience as the sincere are less likely to back down, whereas the insincere will more quickly hop to a different idea.
Making the subject matter public did already harm some people
Hmm, okay. But that, I suggest, appears to have been a case of reasoning oneself stupid.
It does, of course, account for SIAI continuing to attempt to secure the stable doors after the horse has been dancing around in a field for several months taunting them with “COME ON IF YOU THINK YOU’RE HARD ENOUGH.”
(I upvoted XiXiDu’s comment here because he did actually supply a substantive response in PM, well deserving of a vote, and I felt this should be encouraged by reward.)
Roko said he was asked. You didn’t ask him but maybe someone else did?
Roko’s reply to me strongly suggested that he interpreted my message as requesting deletion, and that I was the cause of him deleting it. I doubt anyone at SIAI would have explicitly requested deletion.
I can confirm that I was not asked to delete the comment but did so voluntarily.
I think you are too trigger-happy.
The wording here leaves weird wiggle room—you’re implying it wasn’t Nick?
I’m wondering whether you, Nick, have learned anything from this experience—something perhaps about how attempting to hide something is almost always counterproductive?
Of course, Roko contributed here by deleting the message, you didn’t create this mess by yourself. But you sure have helped. :)
Well, look, I deleted it of my own accord, but only after being prompted that it was a bad thing to have posted. Can we just drop this? It makes me look like even more of a troublemaker than I already look like, and all I really want to do is finish the efficient charity competition then get on with life outside teh intenetz.
This is very important. If the SIAI is the organisation to solve the friendly AI problem and implement CEV then it should be subject to public examination, especially if they ask for money.
The current evidence that anyone anywhere can implement CEV is two papers in six years that talk about it a bit. There appears to have been nothing else from SIAI and no-one else in philosophy appears interested.
If that’s all there is for CEV in six years, and AI is on the order of thirty years away, then (approximately) we’re dead.
This is rather disappointing, as if CEV is possible then a non-artificial general intelligence should be able to implement it, at least partially. And we have those. The reason for CEV is (as I understand it) the danger of the AI going FOOM before it cares about humans. However, human general intelligences don’t go FOOM but should be able to do the work for CEV. If they know what that work is.
Addendum: I see others have been asking “but what do you actually mean?” for a couple of years now.
This strikes me as a demand for particular proof. SIAI is small (and was much smaller until the last year or two), the set of people engaged in FAI research is smaller, Eliezer has chosen to focus on writing about rationality over research for nearly four years, and FAI is a huge problem, in which any specific subproblem should be expected to be underdeveloped at this early stage. And while I and others expect work to speed up in the near future with Eliezer’s attention and better organization, yes, we probably are dead.
Somewhat nitpickingly, this is a reason for FAI in general. CEV is attractive mostly for moving as much work from the designers to the FAI as possible, reducing the potential for uncorrectable error, and being fairer than letting the designers lay out an object-level goal system.
This sounds interesting; do you think you could expand?
It wasn’t intended to be—more incredulity. I thought this was a really important piece of the puzzle, so expected there’d be something at all by now. I appreciate your point: that this is a ridiculously huge problem and SIAI is ridiculously small.
I meant that, as I understand it, CEV is what is fed to the seed AI. Or the AI does the work to ascertain the CEV. It requires an intelligence to ascertain the CEV, but I’d think the ascertaining process would be reasonably set out once we had an intelligence on hand, artificial or no. Or the process to get to the ascertaining process.
I thought we needed the CEV before the AI goes FOOM, because it’s too late after. That implies it doesn’t take a superintelligence to work it out.
Thus: CEV would have to be a process that mere human-level intelligences could apply. That would be a useful process to have, and doesn’t require first creating an AI.
I must point out that my statements on the subject are based in curiosity, ignorance and extrapolation from what little I do know, and I’m asking (probably annoyingly) for more to work with.
“CEV” can (unfortunately) refer to either CEV the process of determining what humans would want if we knew more etc., or the volition of humanity output by running that process. It sounds to me like you’re conflating these. The process is part of the seed AI and is needed before it goes FOOM, but the output naturally is neither, and there’s no guarantee or demand that the process be capable of being executed by humans.
OK. I still don’t understand it, but I now feel my lack of understanding more clearly. Thank you!
(I suppose “what do people really want?” is a large philosophical question, not just undefined but subtle in its lack of definition.)
I have recieved assurances that SIAI will go to significant efforts not to do nasty things, and I believe them. Private assurances given sincerely are, in my opinion, the best we can hope for, and better than we are likely to get from any other entity involved in this.
Besides, I think that XiXiDu, et al are complaining about the difference between cotton and silk, when what is actually likely to happen is more like a big kick in the teeth from reality. SIAI is imperfect. Yes. Well done. Nothing is perfect. At least cut them a bit of slack.
What?!? Open source code—under a permissive license—is the traditional way to signal that you are not going to run off into the sunset with the fruits of a programming effort. Private assurances are usually worth diddly-squat by comparison.
I think that you don’t realize just how bad the situation is. You want that silken sheet. Rude awakening methinks. Also open-source not neccessarily good for FAI in any case.
I don’t think that you realize how bad it is. I’d rather have the universe being paperclipped than supporting the SIAI if that means that I might be tortured for the rest of infinity!
To the best of my knowledge, SIAI has not planned to do anything, under any circumstances, which would increase the probability of you or anyone else being tortured for the rest of infinity.
Supporting SIAI should not, to the best of my knowledge, increase the probability of you or anyone else being tortured for the rest of infinity.
Thank you.
But imagine there was a person a level above yours that went to create some safeguards for an AGI. That person would tell you that you can be sure that the safeguards s/he plans to implement will benefit everyone. Are you just going to believe that? Wouldn’t you be worried and demand that their project is being supervised?
You are in a really powerful position because you are working for an organisation that might influence the future of the universe. Is it really weird to be skeptical and ask for reassurance of their objectives?
Logical rudeness is the error of rejecting an argument for reasons other than disagreement with it. Does your “I don’t think so” mean that you in fact believe that SIAI (possibly) plans to increase the probability of you or someone else being tortured for the rest of eternity? If not, what does this statement mean?
I removed that sentence. I meant that I didn’t believe that the SIAI plans to harm someone deliberately. Although I believe that harm could be a side-effect and that they would rather harm a few beings than allowing some Paperclip maximizer to take over.
You can call me a hypocrite because I’m in favor of animal experiments to support my own survival. But I’m not sure if I’d like to have someone leading an AI project who thinks like me. Take that sentence to reflect my inner conflict. I see why one would favor torture over dust specks but I don’t like such decisions. I’d rather have the universe to end now, or having everyone turned into paperclips, than having to torture beings (especially if I am the being).
I feel uncomfortable that I don’t know what will happen because there is a policy of censorship being favored when it comes to certain thought experiments. I believe that even given negative consequences, transparency is the way to go here. If the stakes are this high, people who believe will do anything to get what they want. That Yudkowsky claims that they are working for the benefit of humanity doesn’t mean it is true. Surely I’d write that and many articles and papers that make it appear this way, if I wanted to shape the future to my liking.
I apologize. I realized my stupidity in interpreting your comment a few seconds after posting the reply (which I then deleted).
Better yet, you could use a kind of doublethink—and then even actually mean it. Here is W. D. Hamilton on that topic:
Discriminating Nepotism—as reprinted in: Narrow Roads of Gene Land, Volume 2 Evolution of Sex, p.356.
In TURING’S CATHEDRAL, George Dyson writes:
I think many people would like to be in that group—if they can find a way to arrange it.
Unless AI was given that outcome (cheerful, contented people etc) as a terminal goal, or that circle of people was the best possible route to some other terminal goal, both of which are staggeringly unlikely, Dyson suspects wrongly.
If you think he suspects rightly, I would really like to see a justification. Keep in mind that AGIs are currently not being built using multi-agent environment evolutionary methods, so any kind of ‘social cooperation’ mechanism will not arise.
Machine intelligence programmers seem likely to construct their machines so as to help them satisfy their preferences—which in turn is likely to make them satisfied. I am not sure what you are talking about—but surely this kind of thing is already happening all the time—with Sergey Brin, James Harris Simons—and so on.
That doesn’t really strike me as a stunning insight, though. I have a feeling that I could find many people who would like to be in almost any group of “cheerful, contented, intellectually and physically well-nourished people.”
This all depends on what the AI wants. Without some idea of its utility function, can we really speculate? And if we speculate, we should note those assumptions. People often think of an AI as being essentially human-like in its values, which is problematic.
It’s a fair description of today’s more successful IT companies. The most obvious extrapolation for the immediate future involves more of the same—but with even greater wealth and power inequalities. However, I would certainly also council caution if extrapolating this out more than 20 years or so.
Currently, there are no entities in physical existence which, to my knowledge, have the ability to torture anyone for the rest of eternity.
You intend to build an entity which would have that ability (or if not for infinity, for a googolplex of subjective years).
You intend to give it a morality based on the massed wishes of humanity—and I have noticed that other people don’t always have my best interests at heart. It is possible—though unlikely—that I might so irritate the rest of humanity that they wish me to be tortured forever.
Therefore, you are, by your own statements, raising the risk of my infinite torture from zero to a tiny non-zero probability. It may well be that you are also raising my expected reward enough for that to be more than counterbalanced, but that’s not what you’re saying—any support for SIAI will, unless I’m completely misunderstanding, raise the probability of infinite torture for some individuals.
See the “Last Judge” section of the CEV paper.
As Vladimir observes, the alternative to SIAI doesn’t involve nothing new happening.
That just pushes the problem along a step. IF the Last Judge can’t be mistaken about the results of the AI running AND the Last Judge is willing to sacrifice the utility of the mass of humanity (including hirself) to protect one or more people from being tortured, then it’s safe. That’s very far from saying there’s a zero probability.
If the Last Judge peeks at the output and finds that it’s going to decide to torture people, that doesn’t imply abandoning FAI, it just requires fixing the bug and trying again.
Just because AGIs have capability to inflict infinite torture, doesn’t mean they have a motive. Also, status quo (with regard to SIAI’s activity) doesn’t involve nothing new happening.
I explained that he is planning to supply one with a possible motive (namely that the CEV of humanity might hate me or people like me). It is precisely because of this that the problem arises. A paperclipper, or any other AGI whose utility function had nothing to do with humanity’s wishes, would have far less motive to do this—it might kill me, but it really would have no motive to torture me.
You can have your private assurances—and I will have my open-source software.
Gollum gave his private assurances to Frodo—and we all know how that turned out.
If someone solicits for you to “trust in me”, alarm bells should start ringing immediately. If you really think that is “the best we can hope for”, then perhaps revisit that.
Well I’m convinced. Frodo should definitely have worked out a way to clone the ring and made sure the information was available to all of Middle Earth. You can never have too many potential Ring-Wraiths.
Suddenly I have a mental image of “The Lord of the Rings: The Methods of Rationality.”
Someone should write that (with a better title). We could have a whole genre of rational fanfiction.
Agreed; Lord of the Rings seems like a natural candidate for discussing AI and related topics.
I’d also like to see His Dark Materials with rationalist!Lyra. The girl had an alethiometer. She should have kicked way more ass than she did as soon as she realized what she had.
Open source AGI is not a good thing. In fact, it would be a disastrously bad thing. Giving people the source code doesn’t just let them inspect it for errors, it also lets them launch it themselves. If you get an AGI close to ready for launch, then sharing its source code means that instead of having one party to decide whether there are enough safety measures ready to launch, you have many parties individually deciding whether to launch it themselves, possibly modifying its utility function to suit their own whim, and the hastiest party’s AGI wins.
Ideally, you’d want to let people study the code, but only trustworthy people, and in a controlled environment where they can’t take the source code with them. But even that is risky, since revealing that you have an AGI makes you a target for espionage and attack by parties who shouldn’t be trusted with humanity’s future.
Actually it reduces the chance of any party drawing massively ahead of the rest. It acts as an equalising force, by power-sharing. Since one of the main things we want to avoid is a disreputable organisation using machine intelligence to gain an advantage—and sustaining it over a long period of time. Using open-source software helps to defend against that possibility.
Machine intelligence will be a race—but it will be a race, whether participants share code or not.
Having said all that, machine intelligence protected by patents with secret source code on a server somewhere does seem like a reasonably probable outcome.
Only if (a) there is no point at which AGIs “foom”, (b) source code sharing is well enough enforced on everyone that no bad organizations combine open source with refinements that they keep secret for an advantage, (c) competing AIs form a stable power equilibrium at all points along their advancement, and (d) it is impossible to trade off goal system stability for optimization power.
I estimate probabilities of 0.4, 0.2, 0.05, and 0.5 for these hypotheses, respectively.
I disagree with most of that analysis. I assume machine intelligence will catalyse its own creation. I fully expect that some organisations will stick with secret source code. How could the probability of that possibly be as low as 0.8!?!
I figure that use of open source software is more likely to lead to a more even balance of power—and less likely to lead to a corrupt organisation in charge of the planet’s most advanced machine intelligence efforts. That assessment is mostly based on the software industry to date—where many of the worst abuses appear to me to have occurred at the hands of proprietary software vendors.
If you have an unethical open source project, people can just fork it, and make an ethical version. With a closed source project, people don’t have that option—they often have to go with whatever they are given by those in charge of the project.
Nor am I assuming that no team will ever win. If there is to be a winner, we want the best possible lead up. The “trust us” model is not it—not by a long shot.
There are two problems with this reasoning. First, you have the causality backwards: makers of open-source software are less abusive than makers of closed-source software not because open-source is such a good safeguard, but because the sorts of organizations that would be abusive don’t open source in the first place.
And second, if there is an unethical AI running somewhere, then forking the code will not save humanity. Forking is a defense against not having good software to use yourself; it is not a defense against other people running software that does bad things to you.
Really? I just provided an example of a mechanism that helps keep open source software projects ethical—the fact that if the manufacturers attempt to exploit their customers it is much easier for the customers to switch to a more ethical fork—because creating such a fork no longer violates copyright law. Though you said you were pointing out problems with my reasoning, you didn’t actually point out any problems with that reasoning.
We saw an example of this kind of thing very recently—with LibreOffice. The developers got afraid that their adopted custodian, Oracle, was going to screw the customers of their project—so, to protect their customers and themselves, they forked it—and went their own way.
If other people are running software that does bad things to you then running good quality software yourself most certainly is a kind of defense. It means you are better able to construct defenses, better able to anticipate their attacks—and so on. Better brains makes you more powerful.
Compare with the closed-source alternative: If other people are running software that does bad things to you—and you have no way to run such software yourself—since it is on their server and running secret source that is also protected by copyright law—you are probably pretty screwed.
How so? I’ve just reread some of your comments on your now deleted post. It looks like you honestly tried to get the SIAI to put safeguards into CEV. Given that the idea spread to many people by now, don’t you think it would be acceptably to discuss the matter before one or more people take it serious or even consider to implement it deliberately?
I don’t think it is a good idea to discuss it. I think that the costs outweigh the benefits. The costs are very big. Benefits marginal.
Ok by me. It is pretty obvious by this point that there is no evil conspiracy involved here. But I think the lesson remains, I you delete something, even if it is just because you regret posting it, you create more confusion than you remove.
I think the question you should be asking is less about evil conspiracies, and more about what kind of organization SIAI is—what would they tell you about, and what would they lie to you about.
If the forbidden topic would be made public (and people would believe it), it would result in a steep rise of donations towards the SIAI. That alone is enough to conclude that the SIAI is not trying to hold back something that would discredit it as an organisation concerned with charitable objectives. The censoring of the information was in accordance with their goal of trying to prevent unfriendly artificial intelligence. Making the subject matter public did already harm some people and could harm people in future.
But the forbidden topic is already public. All the effects that would follow from it being public would already follow. THE HORSE HAS BOLTED. It’s entirely unclear to me what pretending it hasn’t does for the problem or the credibility of the SIAI.
It is not as public as you think. If it was then people like waitingforgodel wouldn’t ask about it.
I’m just trying to figure out how to behave without being able talk about it directly. It’s also really interesting on many levels.
Rather more public than a long forgotten counterfactual discussion collecting dust in the blog’s history books would be. :P
Precisely. The place to hide a needle is in a large stack of needles.
The choice here was between “bad” and “worse”—a trolley problem, a lose-lose hypothetical—and they appear to have chosen “worse”.
I prefer to outsource my needle-keeping security to Clippy in exchange for allowing certain ‘bending’ liberties from time to time. :)
Upvoted for LOL value. We’ll tell Clippy the terrible, no good, very bad idea with reasons as to why this would hamper the production of paperclips.
“Hi! I see you’ve accidentally the whole uFAI! Would you like help turning it into paperclips?”
Brilliant.
Frankly, Clippy would be better than the Forbidden Idea. At least Clippy just wants paperclips.
Of course, if Clippy were clever he would then offer to sell SIAI a commitment to never release the UFAI in exchange for a commitment to produce a fixed number of paperclips per year, in perpetuity.
Admittedly, his mastery of human signaling probably isn’t nuanced enough to prevent that from sounding like blackmail.
I really don’t see how that follows. Will more of the public take it seriously? As I have noted, so far the reaction from people outside SIAI/LW has been “They did WHAT? Are they IDIOTS?”
That doesn’t make it not stupid or not counterproductive. Sincere stupidity is not less stupid than insincere stupidity. Indeed, sincere stupidity is more problematic in my experience as the sincere are less likely to back down, whereas the insincere will more quickly hop to a different idea.
Citation needed.
Citation needed.
I sent you another PM.
Hmm, okay. But that, I suggest, appears to have been a case of reasoning oneself stupid.
It does, of course, account for SIAI continuing to attempt to secure the stable doors after the horse has been dancing around in a field for several months taunting them with “COME ON IF YOU THINK YOU’RE HARD ENOUGH.”
(I upvoted XiXiDu’s comment here because he did actually supply a substantive response in PM, well deserving of a vote, and I felt this should be encouraged by reward.)