Who’s likely to write the AI? A hypothesis
The idea of creating an AI seems to be getting more common.
To the extent that creating an AI is made easier by having more resources rather than by having more carefully thought out philosophy, the first AI will be created by a government or a business, not SIAI. I think the more resources side is the way to bet, but I’m open to argument.
If this is correct, the best strategy for Friendliness may be to keep working on the philosophy but not expect to code, and publicize the risks of Unfriendliness, both seriously and humorously.
The latter is based on something Scott Adams said (for what that’s worth)-- that no one ever realizes they’re the pointy-haired boss, but if anyone says “that plan sounds like something out of Dilbert”, the plan is immediately taken out of consideration.
The good news, such as it is, is that the mistakes likely to be made by corporations and governments can be presented as funnier (or at least more entertaining to people who already dislike those institutions) than those likely to be made by people who are unthinkingly trying to create utopia.
ETA: It’s conceivable that a large organization could have SIAI folks heading its AI project, but this doesn’t seem likely.
I’ll give 50% odds of SIAI being subsumed by a larger organization sometime in the next 10 years.
I’m far from being an expert and I don’t mean to be derogatory either but I still have this feeling that it is an absurd thought that the SIAI could even approach a narrow AI. I wish someone could write some kind of introductory post that would show that the SIAI can indeed accomplish something like a mathematical definition of friendliness given a few people equipped with laptops. A paragraph in a post by multifoliaterose (a mathematician) does express what I feel:
Nobody is able to even define mathematically what a human being is supposed to be, what about friendliness then? Sounds like magic to me. I hope such thoughts are due to my ignorance of math and general AI research.
ETA I’m talking about something like IBM Watson by ‘narrow AI’, not some abstract model of intelligence like AIXI which is about as useless regarding real-world AGI as an universal turing machine when it comes to manufacturing efficient computers (as far as I can tell).
IANAAIR, but I doubt anyone at the SIAI would try to program a way of identifying humans directly into an AI. Instead, it would learn what a human is from observations, using its general learning algorithms. Obviously there are many possible ambiguous cases, but we should try to prevent the AI from needing to make these decisions until it has figured out what goals we want it to have.
It’s not known to be probable, but it’s not known to be impossible either, and the only way to mine this possibility is to make an actual effort.
An existing machine—in this “Volvo S60 - Pedestrian Detection” advert—seems to identify humans, and behave nicely towards them.
Here is more:
That’s nice but probably not what multifoliaterose had in mind. The company has 200 employees and was founded in 1999. I’d be impressed if the SIAI managed to come up with anything nearly as sophisticated. I suppose such companies would call CEV better science fiction. Or do you think they would hire Yudkowsky, if he wanted to get hired? I’m not sure if he is skilled enough to work for such a company. Have you seen any proof of his math or programming skills that would allow you to conclude that he would be able to come up with such a pedestrian detection software, let alone friendliness? (ETA I believe such questions are justified, after all it is important to assess the ability of the SIAI.)
I am a programmer, and I for one, do not see a very strong connection between the potential for building an AGI and programming ability. An AGI isn’t going to come about because you made a really sweet sorting algorithm, it’s going to come about because you had a key insight about what thought is (or something along those lines). 1337 programming skillz probably doesn’t help a lot with that.
Agreed. AGI requires Judea Pearl more than it requires John Carmack.
AGI requires John von Neumann or Alan Turing or the like. Any of them would have decent programming expertise today.
The AGI requires something, that would also result in acquisition of familiarity with the tool-set of mankind, including the actual use of computers for reasoning, which requires you to be able to program. It is enough that the programming expertise might be useful, for the upcoming AGI insight maker to become a good programmer.
Do you think that intelligence is going to be quite simple with hindsight? Something like Einstein’s mass–energy equivalence formula? Because if it is ‘modularly’ then I don’t see how programmers, or mathematicians, won’t be instrumental in making progress towards AGI. Take for example IBM Watson:
It needs a company like IBM to design such a narrow AI. More than 100 algorithms. Could it have been done without a lot of computational and intellectual resources? Can progress be made without tapping into the workings of the human brain, without designing specially optimized hardware, without programming and debugging? Does it really only need some smarts and contemplation to come up with a few key insights to get something that can take over the universe? I’d be interested to learn on how one can arrive at that conclusion.
I agree mathematicians are likely to useful in making AGI. If the folks as SIAI were terrible at math, that would be a bad sign indeed.
I wouldn’t say ‘simple’ but, I would be surprised if it were complex in the same way that Watson is complex. Watson is complex because statistical algorithms can be complex, and Watson has a lot of them. As far as I can tell, there’s nothing conceptually revolutionary about Watson, it’s just a neat and impressive statistical application. I don’t see a strong relationship between Watson-like narrow AI and the goal of AGI.
An AGI might have a lot of algorithms (because intelligence turns out to have a lot of separate components), but the difficulty will be understanding the nature of intelligence and coming up with algorithms and proving the important properties about those algorithms. I wouldn’t expect “practical implementation” to be a separate step where you need programmers because I would expect everything to be implemented in some kind of proof environment.
I too am rather sceptical about the Singularity Institute’s programming skills.
LessWrong itself seems like a nice hack—and there have been some bits and pieces in Java—but hardly enough to inspire confidence in their ability to write reliable and safe software.
Perhaps—if they get sufficient money from donations—they will be able to hire some expert programmers.
They do seem to be pissing off some of the community of experts in the artificial intelligence community a little, though—with their fear-mongering marketing strategy.
Will these groups kiss and make up—if and when the time finally comes to getting down to the business of coding? Or will they remain antagonistic. Stay tuned—I figure.
Hiring expert programmers is trivially easy compared to making the conceptual breakthroughs required for AGI and Friendly AI.
Making a general-purpose intelligent agent is mostly a software engineering problem.
It will probably take programmers to solve it—since they are the most experienced at solving such problems.
That doesn’t solve the problem of what to do with such an agent, of course. That is not so much a software engineering problem.
Not coded by the SIAI as far as I know.
Not that I think it matters much. If you want good programmers you hire them. That part is trivial.
As long as you are or have a skilled enough programmer to identify skilled programmers. My experience working as a programmer suggests that this skill is damn hard to cultivate without just being a good programmer yourself.
My impression is that Eliezer is sufficiently capable that he would be able to identify skilled programmers (I can not guess about others in SIAI). More importantly he is sufficiently well connected that he would be able to hire someone who is fairly well recognised as an expert programmer specifically to consult for the task of identifying and recruiting less well known but more available talent to work at SIAI.
Someone like, say Paul Graham (apparently a skilled programmer, definitely skilled at identifying talent) could perhaps be persuaded to consult.
But the real point is that the task of getting skilled programmers pales in comparison to the task of find people to do AI theory. That is really hard. Implementing the results of the theory is child’s play by comparison.
At the risk of repeating myself, I’m not someone who can’t (ETA I meant can not) really judge this. I have my doubts however that AI theory is a conceptual problem that can be solved by thinking about it. My guess is that you’ll have to do hard science to make progress on the subject. Do research on the human brain, run large scale experiments on supercomputers, build or invent specially optimized hardware, coding and debugging and so on. That it just needs some smarts to come up with a few key insights, I don’t see that. How do people arrive at that conclusion?
The task of getting skilled programmers is probably part of the solution, because resources, intellectual and otherwise, might be instrumental to make progress on the problem.
It is amazing how much difference words like ‘just’ and ‘a few’ can make! This is an extremely hard problem. All sorts of other skills are required but those skills are commodities. They already exist, people have them, you buy them.
What is required to solve something like AIs that are stable when upgrading is extremely intelligent individuals studying the best work in several related fields full time for 15 years… and then having ‘a few insights’.
I think that XiXiDu’s point is that the theory and implementation can not cleanly be divorced. You may need to be constantly programming and trying out the ideas your theory spit out in order to guide and shape the theory to its final, correct form. We can’t necessarily just wait until the theory is developed and then buy the available skill needed to implement it.
Note that the full senctence was:
Well nobody really knows, one way or the other.
So far, machine intelligence has mostly been 60 years of sluggish, gradual progress. On the other hand, we have a pretty neat theory of forecasting—which is the guts of the problem. Maybe we have been doing it all wrong—and there’s a silver bullet that nobody has stumbled across yet.
Hiring good programmers without being a good programmer is far from trivial.
I think the SIAI folk have enough knowledge to deal with this, and there are enough good programmers associated with LW to help if needed. I know for fact cousin_it built some cool software, I’m sure there are others.
I assume that as genetic and bio-engineering and non-organic augmentation come into play, recognizing humans is going to get harder.
I think there might some slight difference in quality in the problems of observing a wide volume of physical reality by varying means and in varying circumstances and unerringly recognizing certain persistent physical processes as corresponding to agents with moral standing such as living, conscious humans, in order to formulate open-ended plans to guarantee their ethically preferred mode of continued existence, and that of analyzing a video feed for patterns that roughly correspond to a 2D projection of an average-human-shaped object and doing some pathfinding adjustments to avoid bumping into it.
A pertinent question is what problem a government or business (not including a general AI startup) may wish to solve with a general AI that is not more easily solved by developing a narrow AI. ‘Easy’ here factors in the risk of failure, which will at least be perceived as very high for a general AI project. Governments and businesses may fund basic research into general AI as part of a strategy to exploit high-risk high-reward opportunities, but are unlikely to do it in-house.
One could also try and figure out some prerequisites for a general AI, and see what would lead to them coming into play. So for instance, I’m pretty sure that a general AI is going to have long-term memory. What AIs are going to get long-term memory? A general AI is going to be able to generalize its knowledge across domains, and that’s probably only going to work properly if it can infer causation. What AIs are going to need to do that?
How much of the world do you need to understand to make reliably good investments?
Do you want your investment computer to be smart enough to say “there’s a rather non-obvious huge bubble in the derivatives based on real estate”? Smart enough to convince you when you don’t want to believe it?