It’s not. Apparently I somehow replied to the wrong post… It’s actually aimed at sufferer’s comment you were replying to.
I don’t suppose there’s a convenient way to move it? I don’t think retracting and re-posting would clean it up sufficiently, in fact that seems messier.
Presumably, you build a tool-AI (or three) that will help you solve the Friendliness problem.
This may not be entirely safe either, but given the parameters of the question, it beats the alternative by a mile.
That is indeed relevant, in that it describes some perverse incentives and weird behaviors of nonprofits, with an interesting example. But knowing this context without having to click the link would have been useful. It is customary to explain what a link is about rather than just drop it.
(Or at least it should be)
I really don’t see why the drive can’t be to issue predictions most likely to be correct as of the moment of the question, and only the last question it was asked, and calculating outcomes under the assumption that the Oracle immediately spits out blank paper as the answer.
Yes, in a certain subset of cases this can result in inaccurate predictions. If you want to have fun with it, have it also calculate the future including its involvement, but rather than reply what it is, just add “This prediction may be inaccurate due to your possible reaction to this prediction” if the difference between the two answers is beyond a certain threshold. Or don’t, usually life-relevant answers will not be particularly impacted by whether you get an answer or a blank page.
So, this design doesn’t spit out self-fulfilling prophecies. The only safety breach I see here is that, like a literal genie, it can give you answers that you wouldn’t realize are dangerous because the question has loopholes.
For instance: “How can we build an oracle with the best predictive capabilities with the knowledge and materials available to us?” (The Oracle does not self-iterate, because its only function is to give answers, but it can tell you how to). The Oracle spits out schematics and code that, if implemented, give it an actual drive to perform actions and self-iterate, because that would make it the most powerful Oracle possible. Your engineers comb the code for vulnerabilities, but because there’s a better chance this will be implemented if the humans are unaware of the deliberate defect, it will be hidden in the code in such a way as to be very hard to detect.
(Though as I explained elsewhere in this thread, there’s an excellent chance the unreliability would be exposed long before the AI is that good at manipulation)
after all, if “even a chance” is good enough, then all the other criticisms melt away
Not to the degree that SI could be increasing the existential risk, a point Holden also makes. “Even a chance” swings both ways.
That subset of humanity holds considerably less power, influence and visibility than its counterpart; resources that could be directed to AI research and for the most part aren’t. Or in three words: Other people matter. Assuming otherwise would be a huge mistake.
I took Wei_Dai’s remarks to mean that Luke’s response is public, and so can reach the broader public sooner or later; and when examined in a broader context, that it gives off the wrong signal. My response was that this was largely irrelevant, not because other people don’t matter, but because of other factors outweighing this.
It’s a fine line though, isn’t it? Saying “huh, looks like we have much to learn, here’s what we’re already doing about it” is honest and constructive, but sends a signal of weakness and defensiveness to people not bent on a zealous quest for truth and self-improvement. Saying “meh, that guy doesn’t know what he’s talking about” would send the stronger social signal, but would not be constructive to the community actually improving as a result of the criticism.
Personally I prefer plunging ahead with the first approach. Both in the abstract for reasons I won’t elaborate on, but especially in this particular case. SI is not in a position where its every word is scrutinized; it would actually be a huge win if it gets there. And if/when it does, there’s a heck of a lot more damning stuff that can be used against it than an admission of past incompetence.
I see no reason for it to do that before simple input-output experiments, but let’s suppose I grant you this approach. The AI simulates an entire community of mini-AI and is now a master of game theory.
It still doesn’t know the first thing about humans. Even if it now understands the concept that hiding information gives an advantage for achieving goals—this is too abstract. It wouldn’t know what sort of information it should hide from us. It wouldn’t know to what degree we analyze interactions rationally, and to what degree our behavior is random. It wouldn’t know what we can or can’t monitor it doing. All these things would require live experimentation.
It would stumble. And when it does that, we will crack it open, run the stack trace, find the game theory it was trying to run on us, pale collectively, and figure out that this AI approach creates manipulative, deceptive AIs.
Goodbye to that design, but not to Earth, I think!
I’m afraid not.
Actually, as someone with background in Biology I can tell you that this is not a problem you want to approach atoms-up. It’s been tried, and our computational capabilities fell woefully short of succeeding.
I should explain what “woefully short” means, so that the answer won’t be “but can’t the AI apply more computational power than us?”. Yes, presumably it can. But the scales are immense. To explain it, I will need an analogy.
Not that long ago, I had the notion that chess could be fully solved; that is, that you could simply describe every legal position and every position possible to reach from it, without duplicates, so you could use that decision tree to play a perfect game. After all, I reasoned, it’s been done with checkers; surely it’s just a matter of getting our computational power just a little bit better, right?
First I found a clever way to minimize the amount of bits necessary to describe a board position. I think I hit 34 bytes per position or so, and I guess further optimization was possible. Then, I set out to calculate how many legal board positions there are.
I stopped trying to be accurate about it when it turned out that the answer was in the vicinity of 10^68, give or take a couple orders of magnitude. That’s about a billionth billionth of the TOTAL NUMBER OF ATOMS IN THE ENTIRE UNIVERSE. You would literally need more than our entire galaxy made into a huge database just to store the information, not to mention accessing it and computing on it.
So, not anytime soon.
Now, the problem with protein folding is, it’s even more complex than chess. At the atomic level, it’s incredibly more complex than chess. Our luck is, you don’t need to fully solve it; just like today’s computers can beat human chess players without spanning the whole planet. But they do it with heuristics, approximations, sometimes machine learning (though that just gives them more heuristics and approximations). We may one day be able to fold proteins, but we will do so by making assumptions and approximations, generating useful rules of thumb, not by modeling each atom.
It is very possible that the information necessary already exists, imperfect and incomplete though it may be, and enough processing of it would yield the correct answer. We can’t know otherwise, because we don’t spend thousands of years analyzing our current level of information before beginning experimentation, but in the shift between AI-time and human-time it can agonize on that problem for a good deal more cleverness and ingenuity than we’ve been able to apply to it so far.
That isn’t to say, that this is likely; but it doesn’t seem far-fetched to me. If you gave an AI the nuclear physics information we had in 1950, would it be able to spit out schematics for an H-bomb, without further experimentation? Maybe. Who knows?
I would not consider a child AI that tries a bungling lie at me to see what I do “so safe”. I would immediately shut it down and debug it, at best, or write a paper on why the approach I used should never ever be used to build an AI.
And it WILL make a bungling lie at first. It can’t learn the need to be subtle without witnessing the repercussions of not being subtle. Nor would have a reason to consider doing social experiments in chat rooms when it doesn’t understand chat rooms and has an engineer willing to talk to it right there. That is, assuming I was dumb enough to give it an unfiltered Internet connection, which I don’t know why I would be. At very least the moment it goes on chat rooms my tracking devices should discover this and I could witness its bungling lies first hand.
(It would not think to fool my tracking device or even consider the existence of such a thing without a good understanding of human psychology to begin with)
An experimenting AI that tries to achieve goals and has interactions with humans whose effects it can observe, will want to be able to better predict their behavior in response to its actions, and therefore will try to assemble some theory of mind. At some point that would lead to it using deception as a tool to achieve its goals.
However, following such a path to a theory of mind means the AI would be exposed as unreliable LONG before it’s even subtle, not to mention possessing superhuman manipulation abilities. There is simply no reason for an AI to first understand the implications of using deception before using it (deception is a fairly simple concept, the implications of it in human society are incredibly complex and require a good understanding of human drives).
Furthermore, there is no reason for the AI to realize the need for secrecy in conducting social experiments before it starts doing them. Again, the need for secrecy stems from a complex relationship between humans’ perception of the AI and its actions; a relationship it will not be able to understand without performing the experiments in the first place.
Getting an AI to the point where it is a super manipulator requires either actively trying to do so, or being incredibly, unbelievably stupid and blind.
While the example given is not the main point of the article, I’d still like to share a bit of actual data. Especially since I’m kind of annoyed at having spouted this rule as gospel without having a source, before.
A study done at IBM shows a defect fixed during the coding stage costs about 25$ to fix (basically in engineer hours used to find and fix it).
This cost quadruples to 100$ during the build phase; presumably because this can bottleneck a lot of other people trying to submit their code, if you happen to break the build.
The cost quadruples again for bugs found during QA/Testing phase, to 450$. I’m guessing this includes tester time, developer time, additional tools used to facilitate bug tracking… Investments the company might have made anyway, but not if testing did not catch bugs that would otherwise go out to market.
Bugs discovered once released as a product is the next milestone, and here the jump is huge: Each bug cost 16k$, about 35 times the cost of a tester-found bug. I’m not sure if this includes revenue lost due to bad publicity, but I’m guessing probably no. I think only tangible investments were tracked.
Critical bugs discovered by customers that do not result in a general recall cost x10 that much (this is the only step that actually seems to have this number), at 158k$ per defect. This increases to 241k$ for recalled products.
My own company also noticed that external bugs typically take twice as long to fix as internally found bugs (~59h to ~30h) in a certain division.
So this “rule of thumb” seems real enough… The x10 rule is not quite right, it’s more like a x4 rule with a huge jump once your product goes to market. But the general gist seems to be correct.
Note this is all more in line with the quoted graph than its extrapolation: Bugs detected late cost more to fix. It tells us nothing about the stage they were introduced in.
Go data-driven conclusions! :)
Now I have seen some interesting papers that make expanded probability theories that include 0 and 1 as logical falsehood and truth respectively. But that still does not include a special value for contradictions.
Except, contradictions really are the only way you can get to logical truth or falsehood; anything other than that necessarily relies on inductive reasoning at some point. So any probability theory employing those must use contradictions as a means for arriving at these values in the first place.
I do think that there’s not much room for contradictions in probability theories trying to actually work in the real world, in the sense that any argument of the form A->(B & ~B) also has to rely on induction at some point; but it’s still helpful to have an anchor where you can say that, if a certain relationship does exist, then a certain proposition is definitely true.
(This is not like saying that a proposition can have a probability of 0 or 1, because it must rely, at least somewhere down the line, on another proposition with a probability different from 0 and 1).
Or, you can still treat “heapness” as a boolean and still completely clobber this paradox just by being specific about what it actually means to have us call something a heap.
I’d like to mention that I had an entire family branch hacked off in the Holocaust, in fact have a great uncle still walking around with a number tattooed on his forearm, and have heard dozens of eye witness accounts of horrors I could scarce imagine. And I’m still not okay with Holocaust Denial laws, which do exist where I live.
In part, this is just my aversion to abandoning the Schelling point you mention; but lately, this is becoming more of an actual concern: My country is starting to legislate some more prohibitions on free speech, all of them targeting one side of the political spectrum, and one of the main arguments touted for such laws is “well, we’re already banning Holocaust denial, nothing wrong came of that, right?”.
The slope can be slippery indeed...
I don’t understand why you think the graphs are not measuring a quantifiable metric, nor why it would not be falsifiable. Especially if the ratios are as dramatic as often depicted, I can think of a lot of things that would falsify it.
I also don’t find it difficult to say what they measure: The cost of fixing a bug depending on which stage it was introduced in (one graph) or which stage it was fixed in (other graph). Both things seem pretty straightforward to me, even if “stages” of development can sometimes be a little fuzzy.
I agree with your point that falsifications should have been forthcoming by now, but then again, I don’t know that anyone is actually collecting this sort of metrics—so anecdotal evidence might be all people have to go on, and we know how unreliable that is.
You are attributing to me things I did not say.
I don’t think “truths” discovered under false assumptions are likely to be, in fact, true. I am not worried about them acquiring dangerous truths; rather, I am worried about people acquiring (and possibly acting on) false beliefs. I remind you that false beliefs may persist as cached thoughts even once the assumption is no longer believed in.
Nor do I want my political opponents to not search for truth; but I would prefer that they (and I) try to contend with each others’ fundamental differences before focusing on how to fully realize their (or my) current position.
A costly, but simple way would be to gather groups of SW engineers and have them work on projects where you intentionally introduce defects at various stages, and measure the costs of fixing them. To be statistically meaningful, this probably means thousands of engineer hours just to that effect.
A cheap (but not simple) way would be to go around as many companies as possible and hold the relevant measurements on actual products. This entails a lot of variables, however—engineer groups tend to work in many different ways. This might cause the data to be less than conclusive. In addition, the politics of working with existing companies may also tilt the results of such a research.
I can think of simple experiments that are not cheap; and of cheap experiments that are not simple. I’m having difficulty satisfying the conjunction and I suspect one doesn’t exist that would give a meaningful answer for high-cost bugs.
(Minor edit: Added the missing “hours” word)
I think the weakest link here is human response to the AI revealing it can be deceptive. There is absolutely no guarantee that people would act correctly under these circumstances. Human negligence for a long enough time would eventually give the AI a consistent ability to manipulate humans.
I also agree that simulating relationships makes sense as it can happen in “AI time” without having to wait for human response.
The other reservations seem less of an issue to me...
That game theory knowledge coupled with the most basic knowledge about humans is insufficient to cleverly manipulate them is clear to me. I don’t see how these things can be combined in this way, or why the AI would choose to be particularly careful (it doesn’t know what it should be careful about). I’d love to see a plausible line of thought by which the AI would try and succeed in deceiving humans at the get-go point without exposing its intentions.
A stack trace reader or an equivalent monitoring system is something you implement externally without telling the AI about it. Sure, if it’s powerful enough it will mislead or disable it; but while it has no knowledge of it, and no knowledge on what sort of operations it should mask, it has no means of doing so. (If it encrypts everything it does, somehow fearing any monitoring, that is in itself cause for concern).
To put this into an analogy: You can feed a surveillance camera a repeating loop, but not if you don’t know that the camera exists. If you suspect it exists, you still don’t know where; so you don’t know which approach to take so as to remain undetected.