Which are the useful areas of AI study?
I’m stuck wondering on a peculiar question lately—which are the useful areas of AI study? What got me thinking is the opinion occasionally stated (or implied) by Eliezer here that performing general AI research might likely have negative utility, due to indirectly facilitating a chance of unfriendly AI being developed. I’ve been chewing on the implications of this for quite a while, as acceptance of these arguments would require quite a change in my behavior.
Right now I’m about to start my CompSci PhD studies soon, and had initially planned to focus on unsupervised domain-specific knowledge extraction from the internet, as my current research background is mostly with narrow AI issues in computational linguistics, such as machine-learning, formation of concepts and semantics extraction. However, in the last year my expectations of singularity and existential risks of unfriendly AI have lead me to believe that focusing my efforts on Friendly AI concepts would be a more valuable choice; as a few years of studies in the area would increase the chance of me making some positive contribution later on.
What is your opinion?
Do studies of general AI topics and research in the area carry a positive or negative utility ? What are the research topics that would be of use to Friendly AI, but still are narrow and shallow enough to make some measurable progress by a single individual/tiny team in the course of a few years of PhD thesis preparation? Are there specific research areas that should be better avoided until more progress has been made on Friendliness research ?
The human race thanks you for considering her in your career path.
I would not worry at all about developing narrow AI. That strikes me as generally positive utility, since you don’t have to solve the hard problem of giving it an enduring general purpose (as you can just give it a fixed narrow one).
An narrow AI employed with solving problems in molecular nanotechnology could be an existential risk nonetheless. It is just a question of scope and control. If it can access enough resources and if humans are sufficiently reckless in implenting whatever it comes up with, then you could end up with runaway real world MNT (if possible at all):
...and...
Just look at what genetic algorithms and evolutionary computation can already do:
Another example:
So what could possible happen if you add some machine intelligence and a bunch of irrational and reckless humans?
That strikes me as mostly the risks inherent in molecular nanotech; the AI isn’t the problematic part. For example, is anything going to go wrong because a GA is optimizing satellite paths?
“Davidson 1997”
This seems to be a reference to a NewScientist article. Is this anything more credible to back up the claim that a GA managed to design a device whose workings human engineers have not managed to understand?
My current guess at FAI-relevant material: Recommended Reading for Friendly AI Research. In short, develop further the kinds of decision theory discussed on LW.
Compression, compression, compression! (in answer to the title).
http://timtyler.org/machine_forecasting/
http://timtyler.org/sequence_prediction/
http://cs.fit.edu/~mmahoney/compression/rationale.html
Others might be better placed to comment on the implications for risk.
On the one hand, having forecasting systems first will be a great boon. It will produce knowledge of the consequences of actions without very much in the way of imposing different values on us.
Also, they illustrate what utility function engineering will look like. I had previously feared people doing utility engineering using reinforcement learning. If you have a forecasting component, that starts to look unnecessary—utility engineering looks set to involve evaluating predicted sensory data instead.
On the less positive side, forecasing systems look easier to build than systems with their own values and tree pruning systems. However, once they are built, the floodgates seem very likely to open. Some will paint that as being bad news.
This seems too nonspecific. Of course an AI can separate the interesting degrees of freedom and ignore the others. The question is how to do this.
How to compress is one question. How to tree-prune is a second, and what values machines should have is a third.
What I didn’t realise—until fairly recently—is the extent to which the first problem is key. It appears that can be usefully be factored out and solved independently.
General-purpose stream compression is a fairly specific computer science problem. However, it is true that we still need to figure out how to do it well.
Having a very detailed understanding of general patterns in English gives you almost as much compression of a textbook as the additional ability to fully understand the subject matter. How does progress on the first ability get us closer to GAI?
You need general purpose compression—not just text compression—if you are being general.
Matt explains why compression is significant here.
Obviously the quality of the model depends on the degree of compression. Poorer compression results in reduced understanding of the material.
In the link, he proves that optimal compression is equivalent to AI. He gives no reason why improving compression is the best approach to developing AI. The OP asked what areas of AI study are useful, not which ones would work in principle. You may have evidence that this is the best approach in practice, but you have not presented it yet.
I don’t actually think that compression is equivalent to intelligence. However, it is pretty close!
My main presentation in that area is in the first of the links I gave:
http://timtyler.org/machine_forecasting/
Brief summary:
I don’t know that it is the best approach in practice—just that it looks like a pretty promising one.
It is not clear to me that what we are measuring is progress. We are definitely improving something, but that does not necessarily get us closer to GAI. Different algorithms could have very different compression efficiencies on different types of data. Some of these may require real progress toward AI, but many types of data can be compressed significantly with little intelligence. A program that could compress any type of non-random data could be improved significantly just by focusing on thing that are easy to predict.
It is not clear to me that what we are measuring is progress.
Compression ratio, on general Occamian data defined w.r.t a small reference machine. If in doubt, refer to Shane Legg: http://www.vetta.org/publications/
Not by very much usually, if there is lots of data. Powerful general purpose compressors would actually work well.
Anyway, that is the idea—to build a general purpose Occamian system—not one with lots of preconceptions. Non-Occamian preconceptions don’t buy that much and are not really needed—if you are sufficiently smart and have a little while to look at the world and see what it is like.
Huffman encoding also works very well in many cases. Obviously you cannot compress any compressible data without GAI, but there are no programs in existence that can compress any compressible data and things like Huffman encoding do make numerical progress in compression, but they do not represent any real conceptual progress. Do you know of any compression programs that make conceptual progress toward GAI? If not, then why do you think that focusing on the compression aspect is likely to provide such progress?
Huffman coding dates back to 1952 - it has been replaced by othehr schemes in size-sensitive applications.
Alexander Ratushnyak seems to be making good progress—see: http://prize.hutter1.net/
I realize that Huffman coding is outdated, I was just trying to make the point that compression progress is possible without AI progress.
Do you have Alexander Ratushnyak’s source code? What techniques does he use that teach us something that could be useful for GAI?
If you have a compressor that isn’t part of a machine intelligence, or is part of a non-state of the art machine intelligence, then that is likely to be true.
However, if your compressor is in a state of the art machine intelligence—and it is the primary tool being used to predict the consequences of its actions, then compression progress (smaller or faster), would translate into increased intelligence. It would help the machine to better predict the consequences of its possible actions, and/or to act more quickly.
That is available here.
Alas, delving in is beyond the scope of this comment.
Your arguments seem to be more inside-view than mine, so I will update my estimates in favour of your point.
I got something from you too. One of the problems with a compression-based approach to machine intelligence is that so far, it hasn’t been very popular. There just aren’t very many people working on it.
Compression is a tough traditional software engineering problem. It seems relatively unglamourous—and there are some barriers to entry, in the form of a big mountain of existing work. Building on that work might not be the most direct way towards the goal—but unless you do that, you can’t easily make competitive products in the field.
Sequence prediction (via stream compression) still seems like the number 1 driving problem to me—and a likely path towards the goal—but some of the above points do seem to count against it.
In a similar vein, I have an interest in researching automated experimental mathematics. Is there any opinion on the relative existential threat level of such a pursuit?
Well, one idea closely related to Friendliness, but possibly more up your alley, is CEV.
Perhaps I completely misunderstand, but I take it to be the case that locating the CEV of mankind is essentially a knowledge-extraction problem—you go about it by interviewing a broad sample of mankind to get their opinions, you then synthesize one or more theories explaining their viewpoint(s), and finally you try to convince them to sign-off on your interpretation of what they believe. (There may well be a rinse-and-repeat cycle.) But the difference is that you are dealing here with values, rather than ‘knowledge’.
I think that it would easily merit a PhD. to take some existing work on interactive domain knowledge extraction and to adapt it to the fuzzier fields of ethics and/or aesthetics. A related topic—also quite important in the context of CEV is the construction of models of ethical opinion which are more-or-less independent of the natural language used by the test subjects.
I’m still up in the air regarding Eliezer’s arguments about CEV.
I have all kinds of ugh-factors coming in mind about not-good or at least not-‘PeterisP-good’ issues an aggregate of 6 billion hairless ape opinions would contain.
The ‘Extrapolated’ part is supposed to solve that; but in that sense I’d say that it turns the whole concept of this problem from knowledge extraction to the extrapolation. In my opinion, the difference between the volition of Random Joe and volition of Random Mohammad (forgive me for stereotyping for the sake of a short example) is much smaller than the difference between volition of Random Joe and the extrapolated volition of Random Joe ‘if he knew more, thought faster, was more the person he wishes he was’. Ergo, the idealistic CEV version of ‘asking everyone’ seems a bit futile. I could go into more detail, but in that case that’s probably material for a separate discussion, analyzing the parts of CEV point by point.
As I see it, adherence to the forms of democracy is important primarily for political reasons—it is firstly a process for gaining the consent of mankind to a compromise, and only secondly a technique for locating the ‘best’ compromise (best by some metric).
Also, as I see it, it is a temporary compromise. We don’t just do a single opinion survey and then extrapolate. We institute a constitution guaranteeing that mankind is to be repeatedly consulted as people become accustomed to the Brave New World of immortality, cognitive enhancement, and fun theory.
In that sense, it’s still futile. The whole reason for the discussion is that AI doesn’t really need permission or consent of anyone; the expected result is that AI—either friendly or unfriendly—will have the ability to enforce the goals of its design. Political reasons will be easily satisfied by a project that claims to try CEV/democracy but skips it in practice, as afterwards the political reasons will cease to have power.
Also, a ‘constitution’ matters only if it is within the goal system of a Friendly AI, otherwise it’s not worth the paper it’s written on.
Well, yes. I am assuming that the ‘constitution’ is part of the CEV, and we are both assuming that CEV or something like it is part of the goal system of the Friendly AI.
I wouldn’t say that it is the whole reason for the discussion, though that is the assumption explaining why many people consider it urgent to get the definition of Friendliness right on the first try. Personally, I think that it is a bad assumption—I believe it should be possible to avoid the all-powerful singleton scenario, and create a ‘society’ of slightly less powerful AIs, each of which really does need the permission and consent of its fellows to continue to exist. But a defense of that position also belongs in a separate discussion.
Or—if you are Ben—you could just non-invasively scan their brains!
Thx for the link. It appears that Goertzel’s CAV is closer to what I was talking about than Yudkowsky’s CEV. But I strongly doubt that scanning brains will be the best way to acquire knowledge of mankind’s collective volition. At least if we want to have already extracted the CEV before the Singularity.
My comment on that topic at the time was:
The other variation from around then was Roko’s document:
Bootstrapping Safe AGI Goal Systems—CEV and variants thereof.
I still have considerable difficulty in taking this kind of material seriously.
No, that is most decidedly not the way to coherently extrapolate volition.
Do you have ideas about how you would coherently extrapolate volition, or are you just sure that interviewing isn’t it?
Off the top of my head: Analysing brain scans and well crafted markets.
Not the way to extrapolate, certainly. But perhaps a reasonable way to find a starting point for the extrapolation. To ‘extrapolate’ is definitely not ‘to derive from first principles’.
Yes, that is one of the various things that could be even worse than opinion polls. Random selection from a set of constructible volition functions would be another.
Friendly AI is to real-life AI R&D as telepathic contact with aliens from Zeta Reticuli is to real-life spaceflight R&D. I wouldn’t waste your time on that stuff. (Yes, I’m aware this is going to get downvotes from the faithful.)
Computational linguistics and machine learning is a good combination, substantial chance of producing something useful and a good background even if you decide to switch to something else later; the tools and concepts you’ll learn will be useful in quite a few interesting domains. I recommend going ahead with that.
You got a downvote from me for preemptively dismissing those who disagree with you as “the faithful”.
He isn’t “preemptively dismissing those who disagree with [him] as ‘the faithful.’” He is saying that the faithful will down vote him without commenting on what the non-faithful will do.
Yes he is.
Although this is literally true.
Another downvote specifically for this comment. It is overtly (albeit mildly) antisocial.