The issue is that it is a doomsday cult if one is to expect extreme outlier (on doom belief) who had never done anything notable beyond being a popular blogger, to be the best person to listen to. That is incredibly unlikely situation for a genuine risk. Bonus cultism points for knowing Bayesian inference but not applying it here. Regardless of how real is the AI risk. Regardless of how truly qualified that one outlier may be. It is an incredibly unlikely world-state where the AI risk would be best coming from someone like that. No matter how fucked up is the scientific review process, it is incredibly unlikely that world’s best AI talk is someone’s first notable contribution.
Dmytry
Karma: 816
how is intelligence well specified compared to space travel? We know physics well enough. We know we want to get from point A to point B. The intelligence: we don’t even quite know what do exactly we want from it. We know of some ridiculous towers of exponents slow method, that means precisely nothing.
but brings forward the date by which we must solve it
Does it really? I already explained that if someone makes an automated engineering tool, all users of that tool are at least as powerful as some (U)FAI based upon this engineering tool. Addition of independent will onto tank doesn’t make it suddenly win the war against much larger force of tanks with no independent will.
You are rationalizing the position here. If you actually reason forwards, it is clear that creation of such tools may, instead, be the life-saver when someone who thought he solved morality unleashes some horror upon the world. (Or sometime, hardware gets so good that very simple evolution simulator like systems could self improve to point of super-intelligence by evolving, albeit that is very far off into the future)
Suppose I were to convince you of butterfly effect, and explain that you sneezing could kill people, months later. And suppose you couldn’t think that non sneezing has same probability. You’d be trying real hard not to sneeze, for nothing, avoid the sudden bright lights (if you have sneeze reflex on bright lights), and so on.
The engineering super-intelligences don’t share our values to such profound extent, as to not even share the desire to ‘do something’ in the real world. Even the engineering intelligence inside my own skull, as far as I can feel. I build designs in real life, because I have rent to pay, or because I am not sure enough it will work and don’t trust the internal simulator that I use for design (i.e. imagining) [and that’s because my hardware is very flawed]. This is also the case with all my friends whom are good engineers.
The issue here is that you conflate things into ‘human level AI’. There’s at least three distinct aspects to AI:
1: Engineering, and other problem solving. This is a creation of designs in abstract design space.
2: Will to do something in real world in real time.
3: Morality.
People here see first two as inseparable, while seeing third as unrelated.
Less Wrong has discussed the meme of “SIAI agrees on ideas that most people don’t take seriously? They must be a cult!”
Awesome, it has discussed this particular ‘meme’, to prevalence of viral transmission of which your words seem to imply it attributes it’s identification as cult. Has it, however, discussed good Bayesian reasoning and understood the impact of a statistical fact that even when there is a genuine risk (if there is such risk), it is incredibly unlikely that the person most worth listening to will be lacking both academic credentials and any evidence of rounded knowledge, and also be an extreme outlier on degree of belief? There’s also the NPD diagnostic criteria to consider. The probabilities multiply here into an incredibly low probability of extreme on many parameters relevant to cult identification, for a non-cult. (For cults, they don’t multiply up because there is common cause.)
edit: to spell out details: So you start with prior maybe 0.1 probability that doomsday salvation group is noncult (and that is massive benefit of the doubt right here), then you look at the founder being such incredibly unlikely combination of traits for a non-cult doomsday caution advocate but such a typical founder for a cult—on multitude of parameters—and then you fuzzily do some knee jerk Bayesian reasoning (which however can be perfectly well replicated using a calculator instead of neuronal signals), and you end up virtually certain it is cult. That’s if you can do Bayes without doing it explicitly on calculator. Now, the reason I am here, is that I did not take a good look until very recently because I did not care if you guys are a cult or not—the cults can be interesting to argue with. And EY is not a bad guy at all, don’t take me wrong, he himself understands that he’s risking making a cult, and trying very hard NOT to make a cult. That’s very redeeming. I do feel bad for the guy, he happened to let one odd belief through, and then voila, a cult that he didn’t want. Or a semi cult, with some people in it for cult reasons and some not so much. He happened not to have formal education, or notable accomplishments that are easily to know are challenging (like being an author of some computer vision library or what ever really). He has some ideas. The cult-follower-type people are dragged towards those ideas like flies to food.
It is unclear to me that artificial intelligence adds any risk there, though, that isn’t present from natural stupidity.
Right now, look, so many plastics around us, food additives, and other novel substances. Rising cancer rates even after controlling for age. With all the testing, when you have hundred random things a few bad ones will slip through. Or obesity. This (idiotic solutions) is a problem with technological progress in general.
edit: actually, our all natural intelligence is very prone to quite odd solutions. Say, reproductive drive, secondary sex characteristics, yadda yadda, end result, cosmetic implants. Desire to sell more product, end result, overconsumption. Etc etc.
Yup, we seem safe for the moment because we simply lack the ability to create anything dangerous.
Actually your scenario already happened… Fukushima reactor failure: they used computer modelling to simulate tsunami, it was 1960s, the computers were science woo, and if computer said so, then it was true.
For more subtle cases though—see, the problem is substitution of ‘intellectually omnipotent omniscient entity’ for AI. If the AI tells to assassinate foreign official, nobody’s going to do that; got to be starting the nuclear war via butterfly effect, and that’s pretty much intractable.
There are machine learning techniques like genetic programming that can result in black-box models.
Which are even more prone to outputting crap solutions even without being superintelligent.
I’m assuming that the modelling portion is a black box so you can’t look inside and see why that solution is expected to lead to a reduction in global temperatures.
Let’s just assume that mister president sits on nuclear launch button by accident, shall we?
It isn’t an amazing novel philosophical insight that type-1 agents ‘love’ to solve problems in the wrong way. It is fact of life apparent even in the simplest automated software of that kind. You, of course, also have some pretty visualization of what is the scenario where the parameter was minimized or maximized.
edit: also the answers could be really funny. How do we solve global warming? Okay, just abduct the prime minister of china! That should cool the planet off.
See, that’s what is so incredibly irritating about dealing with people who lack any domain specific knowledge. You can’t ask it, “how can we reduce global temperatures” in the real world.
You can ask it how to make a model out of data, you can ask it what to do to the model so that such and such function decreases, it may try nuking this model (inside the model), and generate such solution. You got to actually put a lot of effort, like replicating it’s in-model actions in real world in mindless manner, for this nuking to happen in real world. (and you’ll also have the model visualization to examine, by the way)
I think the problem is conflating different aspects of intelligence into one variable. The three major groups of aspects are:
1: thought/engineering/problem-solving/etc; it can work entirely within mathematical model. This we are making steady progress at.
2: real-world volition, especially the will to form most accurate beliefs of the world. This we don’t know how to solve, and don’t even need to automate. We ourselves aren’t even a shining example of 2, but generally don’t care so much about that. 2 is a hard philosophical problem.
3: Morals.
Even strongly superhuman 1 by itself is entirely harmless, even if very general within the problem space of 1. 2 without 1 can’t invent anything. The 3 may follow from strong 1 and 2 assuming that AI assigns non zero chance to being under test in a simulation, and strong 1 providing enormous resources.
So, what is your human level AI?
It seems to me that people with high capacity for 1, i.e. the engineers and scientists, are so dubious about AI risk because it is pretty clear to them, both internally, and from the AI effort, that 1 doesn’t imply 2 and adding 2 won’t strengthen 1. There isn’t some great issue with 1 that 2 would resolve. The 1 works just fine. If for example we invent awesome automatic software development AI, it will be harmless even if superhuman at programming, and will self improve as much as possible without 2. Not just harmless, there’s no reason why 1-agent plus human are together any less powerful than 1-agent with 2-capability.
Eliezer, it looks like, is very concerned with forming accurate beliefs, i.e. 2-type behaviour, but i don’t see him inventing novel solutions as much. Maybe he’s so scared of the AI because he attributes other people’s problem solving to intellect paralleling his, while it’s more orthogonal. Maybe he imagines that very strongly more-2 agent will somehow be innovative and foom, and he sees a lot of room for improving the 2. Or something along those lines. He is a very unusual person; I don’t know how he thinks. The way I think it is very natural for me that the problem solving does not require wanting to actually do anything real first. That also parallels the software effort because ultimately everyone who is capable of working effectively as innovative software developers are very 1-orientated and don’t see 2 as either necessary or desirable. I don’t think 2 would just suddenly appear out of nothing by some emergence or accident.
Pretty ordinary meaning: Bunch of people trusting extraordinary claims not backed with any evidence or expert consensus, originating from a charismatic leader who is earning living off cultists. Subtype doomsday. Now, I don’t give any plus or minus points for the leader and living off cultists part, but the general lack of expert concern of the issue is a killer. Experts being people with expertise on relevant subject (but no doomsday experts allowed; has to be something practically useful or at least not all about the doomsday itself. Else you start counting theologians as experts). E.g. for AI risk, the relevant experts may be people with CS accomplishments, the folks who made self driving car, the visual object recognition experts, speech recognition, who developed actual working AI of some kind, etc etc.
I wonder what’d happen if we’d train a SPR for cult recognition. http://lesswrong.com/lw/3gv/statistical_prediction_rules_outperform_expert/ SPRs don’t care for any unusual redeeming qualities or special circumstances.
Can you list some non-cult most similar to LW/SIAI ?
If it starts worrying more than astronomers do, sure. The few is as in percentile, at same level of the worry.
More generally, if the degree of the belief is negatively correlated with achievements in relevant areas of expertise, then the extreme forms of belief are very likely false. (And just in case: comparing to Galileo is cherry picking. For each Galileo there’s a ton of cranks)
Yep. Majorly awesome scenario degrades into ads vs adblock when you consider everything in the future not just the self willed robot. Matter of fact, a lot of work is put into constructing convincing strings of audio and visual stimuli, and into ignoring those strings.
You’re still falling into the same trap, thinking that your work is ok as long as it doesn’t immediately destroy the Earth. What if someone takes your proof generator design, and uses the ideas to build something that does affect the real world?
Well let’s say in 2022 we have a bunch of tools along the lines of automatic problem solving, unburdened by their own will (not because they were so designed but by simple omission of immense counter productive effort). Someone with a bad idea comes around, downloads some open source software, cobbles together some self propelling ‘thing’ that is ‘vastly superhuman’ circa 2012. Keep in mind that we still have our tools that make us ‘vastly superhuman’ circa 2012 , and i frankly don’t see how ‘automatic will’, for lack of better term, is contributing anything here that would make the fully automated system competitive.
Well, there’s this implied assumption that super-intelligence that ‘does not share our values’ shares our domain of definition of the values. I can make a fairly intelligent proof generator, far beyond human capability if given enough CPU time; it won’t share any values with me, not even the domain of applicability; the lack of shared values with it is so profound as to make it not do anything whatsoever in the ‘real world’ that I am concerned with. Even if it was meta—strategic to the point of potential for e.g. search for ways to hack into a mainframe to gain extra resources to do the task ‘sooner’ by wallclock time, it seems very dubious that by mere accident it will have proper symbol grounding, won’t wirelead (i.e. would privilege the solutions that don’t involve just stopping said clock), etc etc. Same goes for other practical AIs, even the evil ones that would e.g. try to take over internet.
I’m kind of dubious that you needed ‘beware of destroying mankind’ in a physics textbook to get Teller to check if nuke can cause thermonuclear ignition in atmosphere or seawater, but if it is there, I guess it won’t hurt.
Choosing between mathematically equivalent interpretations adds 1 bit of complexity that doesn’t need to be added. Now, if EY had derived the Born probabilities from first principles, that’d be quite interesting.
Seems like a prime example of where to apply rationality: what are the consequences to trying to work on AI risk right now? Versus on something else? Does AI risk work have good payoff?
What’s of the historical cases? The one example I know of is this: http://www.fas.org/sgp/othergov/doe/lanl/docs1/00329010.pdf (thermonuclear ignition of atmosphere scenario). Can a bunch of people with little physics related expertise do something about such risks >10 years before? Beyond the usual anti war effort? Bill Gates will work on AI risk when it becomes clear what to do about it.
SI/LW sometimes gives the impression of being a doomsday cult,
Because it fits the pattern exactly. If you have top astronomers worrying about meteorite hitting earth, that is astronomy. If you have nonastronomers (with very few astronomers) worrying about meteorite hitting earth, that’s doomsday cult. Or at very best, a vague doomsday cult. edit: Just saying, that’s how I classify, works for me. If you have instances (excluding SIAI) where this method of classification fails in damaging way, I am very interested to hear of them, to update my classification method. I might be misclassifying something. I might just go through the list of things that i classified as cults, and classify some items on that list as non-cult, if the classification method fails.
The argument is for the insights coming out of EY , and the privileging that EY is making for those hypotheses originated by others, aka cherrypicking what to advertise. EY is a good writer.
edit: concrete thought example: There is a drug A that undergoes many tests, with some of them evaluating it as better than placebo, some as equal to placebo, and some as worse to placebo. Worst of all, each trial is conducted on 1 person’s opinion. Comes in the charismatic pharmaceutical marketer, or charismatic anti-vaccination campaign leader, and starts bringing to attention the negative or positive trials. That is not good. Even if there’s both of those people.