I am one of those who haven’t been convinced by the SIAI line. I have two main objections.
First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies. (This is also the basis of my skepticism about cryonics.) If you’re going to say “Technology X is likely to be developed” then I’d like to see your prediction mechanism and whether it’s worked in the past.
Second, shouldn’t an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there’s room for pure philosophy and mathematics, but you’d need some grounding in actual AI to understand what future AIs are likely to do.
I think multifoliaterose is right that there’s a PR problem, but it’s not just a PR problem. It seems, unfortunately, to be a problem with having enough justification for claims, and a problem with connecting to the world of professional science. I think the PR problems arise from being too disconnected from the demands placed on other scientific or science policy organizations. People who study other risks, say epidemic disease, have to get peer-reviewed, they have to get government funding—their ideas need to pass a round of rigorous criticism. Their PR is better by necessity.
First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies.
As was mentioned in other threads, SIAI’s main arguments rely on disjunctions and antipredictions more than conjunctions and predictions. That is, if several technology scenarios lead to the same broad outcome, that’s a much stronger claim than one very detailed scenario.
For instance, the claim that AI presents a special category of existential risk is supported by such a disjunction. There are several technologies today which we know would be very dangerous with the right clever ‘recipe’– we can make simple molecular nanotech machines, we can engineer custom viruses, we can hack into some very sensitive or essential computer systems, etc. What these all imply is that a much smarter agent with a lot of computing power is a severe existential threat if it chooses to be.
There needs to be an article on this point. In the absence of a really good way of deciding what technologies are likely to be developed, you are still making a decision. You haven’t signed up yet; whether you like it or not, that is a decision. And it’s a decision that only makes sense if you think technology X is unlikely to be developed, so I’d like to see your prediction mechanism and whether it’s worked in the past. In the absence of really good information, we sometimes have to decide on the information we have.
EDIT: I was thinking about cryonics when I wrote this, though the argument generalizes.
My point, with this, is that everybody is risk-averse and everybody has a time preference. The less is known about the prospects of a future technology, the less willing people are to invest resources into ventures that depend on the future development of that technology. (Whether to take advantage of the technology—as in cryonics—or to mitigate its dangers—as in FAI.) Also, the farther in the future the technology is, the less people care about it; we’re not willing to spend much to achieve benefits or forestall risks in the far future.
I don’t think it’s reasonable to expect people to change these ordinary features of economic preference. If you’re going to ask people to chip in to your cause, and the time horizon is too far, or the uncertainty too high, they’re not going to want to spend their resources that way. And they’ll be justified.
Note: yes, there ought to be some magnitude of benefit or cost that overcomes both risk aversion and time preference. Maybe you’re going to argue that existential risk and cryonics are issues of such great magnitude that they outweigh both risk aversion and time preference.
But: first of all, the importance of the benefit or cost is also an unknown (and indeed subjective.) How much do you value being alive? And, second of all, nobody says our risk and time preferences are well-behaved. There may be a date so far in the future that I don’t care about anything that happens then, no matter how good or how bad. There may be loss aversion—an amount of money that I’m not willing to risk losing, no matter how good the upside. I’ve seen some experimental evidence that this is common.
My point, with this, is that everybody is risk-averse and everybody has a time preference.
From what I understand this applies to most people but not everyone, especially outside of contrived laboratory circumstances. Overconfidence and ambition essentially amount to risk-loving choices for some major life choices.
you haven’t signed up yet; whether you like it or not, that is a decision. And it’s a
decision that only makes sense if you think technology X is unlikely to be developed
What is it that is making you think that whatever SarahC hasn’t “signed up” to is having a positive effect—and that she can’t do something better with her resources?
First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies. (This is also the basis of my skepticism about cryonics.) If you’re going to say “Technology X is likely to be developed” then I’d like to see your prediction mechanism and whether it’s worked in the past.
Let’s keep in mind that your estimated probabilities of various technological advancements occurring and your level of confidence in those estimates are completely distinct… In particular, here you seem to express low estimated probabilities of various advancements occurring, and you justify this by saying “we really have no idea”. This seems like a complete non sequitur. Maybe you have a correct argument in your mind, but you’re not giving us all the pieces.
Technology X is likely to be developed in a few decades.
Technology X is risky.
We must take steps to mitigate the risk.
If you haven’t demonstrated 1 -- if it’s still unknown—you can’t expect me to believe 3. The burden of proof is on whoever’s asking for money for a new risk-mitigating venture, to give strong evidence that the risk is real.
So you think a danger needs to likely arrive in a few decades for it to merit attention?
I think that is quite irresponsible. No law of physics states that all problems can certainly be solved very well in a few decades (the solutions for some problems might even necessarily involve political components, btw), so starting preparations earlier can be necessary.
I see “burden of proof” as a misconcept in the same way that someone “deserving” something is. A better way of thinking about this: “You seem to be making a strong claim. Mind sharing the evidence for your claim for me? …I disagree that the evidence you present justifies your claim.”
For what it’s worth, I also see “must _” as a misconcept—although “must _ to _” is not. It’s an understandable usage if the “to _*” clause is implicit, but that doesn’t seem true in this case. So to fix up SIAI’s argument, you could say that these are the statements whose probabilities are being contested:
If SarahC takes action Y before the development of Technology X and Technology X is developed, the expected value of her action will exceed its cost.
Technology X will be developed.
And depending on their probabilities, the following may or may not be true:
SarahC wants to take action Y.
Pretty much anything you say that’s not relevant to one of statements 1 or 2 (including statements that certain people haven’t been “responsible” enough in supporting their claims) is completely irrelevant to the question of whether you want to take action Y. You already have (or ought to be able to construct) probability estimates for each of 1 and 2.
Your grasp of decision theory is rather weak if you are suggesting that when Technology X is developed is irrelevant to SarahC’s decision. Similarly, you seem to suggest that the ratio of value to cost is irrelevant and that all that matters is which is bigger. Wrong again.
But your real point was not to set up a correct decision problem, but rather to suggest that her questions about whether “certain people” have been “responsible” are irrelevant. Well, I have to disagree. If action Y is giving money to “certain people”, then their level of “responsibility” is very relevant.
I did enjoy your observations regarding “burden of proof” and “must”, though probably not as much as you did.
Your grasp of decision theory is rather weak if you are suggesting that when Technology X is developed is irrelevant to SarahC’s decision.
Of course that is important. I didn’t want to include a lot of qualifiers.
I’m not trying to make a bulletproof argument so much as concisely give you an idea of why I think SarahC’s argument is malformed. My thinking is that should be enough for intellectually honest readers, as I don’t have important insights to offer beyond the concise summary. If you think I ought to write longer posts with more qualifications for readers who aren’t good at taking ideas seriously feel free to say that.
Similarly, you seem to suggest that the ratio of value to cost is irrelevant and that all that matters is which is bigger. Wrong again.
Really? So in some circumstances it is rational to take an action for which the expected cost is greater than the expected value? Or it is irrational to take an action for which the expected value exceeds the expected cost? (I’m using “rational” to mean “expected utility maximizing”, “cost” to refer to negative utility, and “value” to refer to positive utility—hopefully at this point my thought process is transparent.)
If action Y is giving money to “certain people”, then their level of “responsibility” is very relevant.
It would be a well-formed argument to say that because SIAI folks make strong claims without justifying them, they won’t use money SarahC donates well. As far as I can tell, SarahC has not explicitly made that argument. (Recall I said that she might have a correct argument in her mind but she isn’t giving us all the pieces.)
I did enjoy your observations regarding “burden of proof” and “must”, though probably not as much as you did.
Please no insults, this isn’t you versus me is it?
Similarly, you seem to suggest that the ratio of value to cost is irrelevant and that all that matters is which is bigger. Wrong again.
Really? So in some circumstances it is rational to take an action for which the expected cost is greater than the expected value?
No, your error was in the other direction. If you look back carefully, you will notice that the ratio is being calculated conditionally on Technology X being developed. Given that the cost is sunk regardless of whether the technology appears, it is possible that SarahC should not act even though the (conditionally) expected return exceeds the cost.
Please no insults, this isn’t you versus me is it?
Shouldn’t be. Nor you against her. I was catty only because I imagined that you were being catty. If you were not, then I surely apologize.
I didn’t say what SarahC should do with the probabilities once she had them. All I said was that they were pretty much all was relevant to the question of whether she should donate. Unless I didn’t, in which case I meant to.
Second, shouldn’t an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there’s room for pure philosophy and mathematics, but you’d need some grounding in actual AI to understand what future AIs are likely to do.
I’m not sure what you refer to by “actual AI.” There is a sub-field of academic computer science which calls itself “Artificial Intelligence,” but it’s not clear that this is anything more than a label, or that this field does anything more than use clever machine learning techniques to make computer programs accomplish things that once seemed to require intelligence (like playing chess, driving a car, etc.)
I’m not sure why it is a requirement that an organization concerned with the behavior of hypothetical future engineered minds would need to be in contact with these researchers.
I’m not sure why it is a requirement that an organization concerned with the behavior of hypothetical future engineered minds would need to be in contact with these researchers.
You have to know some of their math (some of it is interesting, some not) but this does not require getting on the phone with them and asking them to explain their math, to which of course they would tell to you to RTFM instead of calling them.
Yes, the subfield of computer science is what I’m referring to.
I’m not sure that the difference between “clever machine learning techniques” and “minds” is as hard and fast as you make it. A machine that drives a car is doing one of the things a human mind does; it may, in some cases, do it through a process that’s structurally similar to the way the human mind does it. It seems to me that machines that can do these simple cognitive tasks are the best source of evidence we have today about hypothetical future thinking machines.
I’m not sure that the difference between “clever machine learning techniques” and “minds” is as hard and fast as you make it.
I gave the wrong impression here. I actually think that machine learning might be a good framework for thinking about how parts of the brain work, and I am very interested in studying machine learning. But I am skeptical that more than a small minority of projects where machine learning techniques have been applied to solve some concrete problem have shed any light on how (human) intelligence works.
In other words, I largely agree with Ben Goertzel’s assertion that there is a fundamental difference between “narrow AI” and AI research that might eventually lead to machines capable of cognition, but I’m not sure I have good evidence for this argument.
In other words, I largely agree with Ben Goertzel’s assertion that there is a fundamental difference between “narrow AI” and AI research that might eventually lead to machines capable of cognition, but I’m not sure I have good evidence for this argument.
Although one should be very, very careful not to confuse the opinions of someone like Goertzel with those of the people (currently) at SIAI, I think it’s fair to say that most of them (including, in particular, Eliezer) hold a view similar to this. And this is the location—pretty much the only important one—of my disagreement with those folks. (Or, rather, I should say my differing impression from those folks—to make an important distinction brought to my attention by one of the folks in question, Anna Salamon.) Most of Eliezer’s claims about the importance of FAI research seem obviously true to me (to the point where I marvel at the fuss that is regularly made about them), but the one that I have not quite been able to swallow is the notion that AGI is only decades away, as opposed to a century or two. And the reason is essentially disagreement on the above point.
At first glance this may seem puzzling, since, given how much more attention is given to narrow AI by researchers, you might think that someone who believes AGI is “fundamentally different” from narrow AI might be more pessimistic about the prospect of AGI coming soon than someone (like me) who is inclined to suspect that the difference is essentially quantitative. The explanation, however, is that (from what I can tell) the former belief leads Eliezer and others at SIAI to assign (relatively) large amounts of probability mass to the scenario of a small set of people having some “insight” which allows them to suddenly invent AGI in a basement. In other words, they tend to view AGI as something like an unsolved math problem, like those on the Clay Millennium list, whereas it seems to me like a daunting engineering task analogous to colonizing Mars (or maybe Pluto).
This—much more than all the business about fragility of value and recursive self-improvement leading to hard takeoff, which frankly always struck me as pretty obvious, though maybe there is hindsight involved here—is the area of Eliezer’s belief map that, in my opinion, could really use more public, explicit justification.
whereas it seems to me like a daunting engineering task analogous to colonizing Mars
I don’t think this is a good analogy. The problem of colonizing Mars is concrete. You can make a TODO list; you can carve the larger problem up into subproblems like rockets, fuel supply, life support, and so on. Nobody knows how to do that for AI.
OK, but it could still end up being like colonizing Mars if at some point someone realizes how to do that. Maybe komponisto thinks that someone will probably carve AGI in to subproblems before it is solved.
Well, it seems we disagree. Honestly, I see the problem of AGI as the fairly concrete one of assembling an appropriate collection of thousands-to-millions of “narrow AI” subcomponents.
Perhaps another way to put it would be that I suspect the Kolmogorov complexity of any AGI is so high that it’s unlikely that the source code could be stored in a small number of human brains (at least the way the latter currently work).
EDIT: When I say “I suspect” here, of course I mean “my impression is”. I don’t mean to imply that I don’t think this thought has occurred to the people at SIAI (though it might be nice if they could explain why they disagree).
An oddly somewhat relevant article on the information needed for specifying the brain. It is a biologist tearing a strip out of kurzweil for suggesting that we’ll be able reverse engineer the human brain in a decade by looking at the genome.
P.Z. is misreading a quote from a secondhand report. Kurzweil is not talking about reading out the genome and simulating the brain from that, but about using improvements in neuroimaging to inform input-output models of brain regions. The genome point is just an indicator of the limited number of component types involved, which helps to constrain estimates of difficulty.
Edit: Kurzweil has now replied, more or less along the lines above.
Kurzweil’s analysis is simply wrong. Here’s the gist of my refutation of it:
“So, who is right? Does the brain’s design fit into the genome? - or not?
The detailed form of proteins arises from a combination of the nucleotide sequence that specifies them, the cytoplasmic environment in which gene expression takes place, and the laws of physics.
We can safely ignore the contribution of cytoplasmic inheritance—however, the contribution of the laws of physics is harder to discount. At first sight, it may seem simply absurd to argue that the laws of physics contain design information relating to the construction of the human brain. However there is a well-established mechanism by which physical law may do just that—an idea known as the anthropic principle. This argues that the universe we observe must necessarily permit the emergence of intelligent agents. If that involves a coding the design of the brains of intelligent agents into the laws of physics then: so be it. There are plenty of apparently-arbitrary constants in physics where such information could conceivably be encoded: the fine structure constant, the cosmological constant, Planck’s constant—and so on.
At the moment, it is not even possible to bound the quantity of brain-design information so encoded. When we get machine intelligence, we will have an independent estimate of the complexity of the design required to produce an intelligent agent. Alternatively, when we know what the laws of physics are, we may be able to bound the quantity of information encoded by them. However, today neither option is available to us.”
I agree with your analysis, but I also understand where PZ is coming from. You write above that the portion of the genome coding for the brain is small. PZ replies that the small part of the genome you are referring to does not by itself explain the brain; you also need to understand the decoding algorithm—itself scattered through the whole genome and perhaps also the zygotic “epigenome”. You might perhaps clarify that what you were talking about with “small portion of the genome” was the Kolmogorov complexity, so you were already including the decoding algorithm in your estimate.
The problem is, how do you get the point through to PZ and other biologists who come at the question from an evo-devo PoV? I think that someone ought to write a comment correcting PZ, but in order to do so, the commenter would have to speak the languages of three fields—neuroscience, evo-devo, and information-theory. And understand all three well enough to unpack the jargon to laymen without thereby loosing credibility with people who do know one or more of the three fields.
Obviously the genome alone doesn’t build a brain. I wonder how many “bits” I should add on for the normal environment that’s also required (in terms of how much additional complexity is needed to get the first artificial mind that can learn about the world given additional sensory-like inputs). Probably not too many.
Honestly, I see the problem of AGI as the fairly concrete one of assembling an appropriate collection of thousands-to-millions of “narrow AI” subcomponents.
What do you think you know and how do you think you know it? Let’s say you have a thousand narrow AI subcomponents. (Millions = implausible due to genome size, as Carl Shulman points out.) Then what happens, besides “then a miracle occurs”?
What happens is that the machine has so many different abilities (playing chess and walking and making airline reservations and...) that its cumulative effect on its environment is comparable to a human’s or greater; in contrast to the previous version with 900 components, which was only capable of responding to the environment on the level of a chess-playing, web-searching squirrel.
This view arises from what I understand about the “modular” nature of the human brain: we think we’re a single entity that is “flexible enough” to think about lots of different things, but in reality our brains consist of a whole bunch of highly specialized “modules”, each able to do some single specific thing.
Now, to head off the “Fly Q” objection, Iet me point out that I’m not at all suggesting that an AGI has to be designed like a human brain. Instead, I’m “arguing” (expressing my perception) that the human brain’s general intelligence isn’t a miracle: intelligence really is what inevitably happens when you string zillions of neurons together in response to some optimization pressure. And the “zillions” part is crucial.
(Whoever downvoted the grandparent was being needlessly harsh. Why in the world should I self-censor here? I’m just expressing my epistemic state, and I’ve even made it clear that I don’t believe I have information that SIAI folks don’t, or am being more rational than they are.)
If a thousand species in nature with a thousand different abilities were to cooperate, would they equal the capabilities of a human? If not, what else is missing?
Tough problem. My first reaction is ‘yes’, but I think that might be because we’re assuming cooperation, which might be letting more in the door than you want.
I am highly confused about the parent having been voted down, to the point where I am in a state of genuine curiosity about what went through the voter’s mind as he or she saw it.
Eliezer asked whether a thousand different animals cooperating could have the power of a human. I answered:
Yes, if there were a sufficiently powerful optimization process controlling the form of their cooperation
And then someone came along, read this, and thought....what? Was it:
“No, you idiot, obviously no optimization process could be that powerful.” ?
“There you go: ‘sufficiently powerful optimization process’ is equivalent to ‘magic happens’. That’s so obvious that I’m not going to waste my time pointing it out; instead, I’m just going to lower your status with a downvote.” ?
“Clearly you didn’t understand what Eliezer was asking. You’re in over your head, and shouldn’t be discussing this topic.” ?
Do you expect the conglomerate entity to be able to read or to be able to learn how to? Considering Eliezer can quite happily pick many many things like archer fish (ability to shoot water to take out flying insects) and chameleons (ability to control eyes independently), I’m not sure how they all add up to reading.
Natural selection is an optimization process, but it isn’t intelligent.
Also, the point here is AI—one is allowed to assume the use of intelligence in shaping the cooperation. That’s not the same as using intelligence as a black box in describing the nature of it.
If you were the downvoter, might I suggest giving me the benefit of the doubt that I’m up to speed on these kinds of subtleties? (I.e. if I make a comment that sounds dumb to you, think about it a little more before downvoting?)
Now it’s my turn to downvote, on the grounds that you didn’t understand my comment. I agree that natural selection is unintelligent—that was my whole point! It was intended as a counterexample to your implied assertion that an appeal to an optimization process is an appeal to intelligence.
EDIT: I suppose this confirms on a small scale what had become apparent in the larger discussion here about SIAI’s public relations: people really do have more trouble noticing intellectual competence than I tend to realize.
(N.B. I just discovered that I had not, in fact, downvoted the comment that began this discussion. I must have had it confused with another.)
Like Eliezer, I generally think of intelligence and optimization as describing the same phenomenon. So when I saw this exchange:
If a thousand species in nature with a thousand different abilities were to cooperate, would they equal the capabilities of a human? If not, what else is missing?
Yes, if there were a sufficiently powerful optimization process controlling the form of their cooperation.
I read your reply as meaning approximately “1000 small cognitive modules are a really powerful optimization process if and only if their cooperation is controlled by a sufficiently powerful optimization process.”
To answer the question you asked here, I thought the comment was worthy of a downvote (though apparently I did not actually follow through) because it was circular in a non-obvious way that contributed only confusion.
I am probably a much more ruthless downvoter than many other LessWrong posters; my downvotes indicate a desire to see “fewer things like this” with a very low threshold.
I read your reply as meaning approximately “1000 small cognitive modules are a really powerful optimization process if and only if their cooperation is controlled by a sufficiently powerful optimization process.”
Thank you for explaining this, and showing that I was operating under the illusion of transparency.
My intended meaning was nothing so circular. The optimization process I was talking about was the one that would have built the machine, not something that would be “controlling” it from inside. I thought (mistakenly, it appears) that this would be clear from the fact that I said “controlling the form of their cooperation” rather than “controlling their cooperation”. My comment was really nothing different from thomblake’s or wedrifid’s. I was saying, in effect, “yes, on the assumption that the individual components can be made to cooperate, I do believe that it is possible to assemble them in so clever a manner that their cooperation would produce effective intelligence.”
The “cleverness” referred to in the previous sentence is that of the whatever created the machine (which could be actual human programmers, or, theoretically, something else like natural selection) and not the “effective intelligence” of the machine itself. (Think of a programmer, not a homunculus.) Note that I easily envision the process of implementing such “cleverness” itself not looking particularly clever—perhaps the design would be arrived at after many iterations of trial-and-error, with simpler devices of similar form. (Natural selection being the extreme case of this kind of process.) So I’m definitely not thinking magically here, and least not in any obvious way (such as would warrant a downvote, for example).
I can now see how my words weren’t as transparent as I thought, and thank you for drawing this to my attention; at the same time, I hope you’ve updated your prior that a randomly selected comment of mine results from a lack of understanding of basic concepts.
Consider me updated. Thank you for taking my brief and relatively unhelpful comments seriously, and for explaining your intended point. While I disagree that the swiftest route to AGI will involve lots of small modules, it’s a complicated topic with many areas of high uncertainty; I suspect you are at least as informed about the topic as I am, and will be assigning your opinions more credence in the future.
Downvoted for retaliatory downvoting; voted everything else up toward 0.
Downvoted the parent and upvoted the grandparent. “On the grounds that you didn’t understand my comment” is a valid reason for downvoting and based off a clearly correct observation.
I do agree that komponisto would have been better served by leaving off mention of voting altogether. Just “You didn’t understand my comment. …” would have conveyed an appropriate level of assertiveness to make the point. That would have avoided sending a signal of insecurity and denied others the invitation to judge.
Status matters; it’s a basic human desideratum, like food and sex (in addition to being instrumentally useful in various ways). There seems to be a notion among some around here that concern with status is itself inherently irrational or bad in some way. But this is as wrong as saying that concern with money or good-tasting food is inherently irrational or bad. Yes, we don’t want the pursuit of status to interfere with our truth-detecting abilities; but the same goes for the pursuit of food, money, or sex, and no one thinks it’s wrong for aspiring rationalists to pursue those things. Still less is it considered bad to discuss them.
Comments like the parent are disingenuous. If we didn’t want users to think about status, we wouldn’t have adopted a karma system in the first place. A norm of forbidding the discussion of voting creates the wrong incentives: it encourages people to make aggressive status moves against others (downvoting) without explaining themselves. If a downvote is discussed, the person being targeted at least has better opportunity to gain information, rather than simply feeling attacked. They may learn whether their comment was actually stupid, or if instead the downvoter was being stupid. When I vote comments down I usually make a comment explaining why—certainly if I’m voting from 0 to −1. (Exceptions for obvious cases.)
I really don’t appreciate what you’ve done here. A little while ago I considered removing the edit from my original comment that questioned the downvote, but decided against it to preserve the context of the thread. Had I done so I wouldn’t now be suffering the stigma of a comment at −1.
When I vote comments down I usually make a comment explaining why—certainly if I’m voting from 0 to −1. (Exceptions for obvious cases.)
Then you must be making a lot of exceptions, or you don’t downvote very much. I find that “I want to see fewer comments like this one” is true of about 1⁄3 of the comments or so, though I don’t downvote quite that much anymore since there is a cap now. Could you imagine if every 4th comment in ‘recent comments’ was taken up by my explanations of why I downvoted a comment? And then what if people didn’t like my explanations and were following the same norm—we’d quickly become a site where most comments are explaining voting behavior.
A bit of a slippery slope argument, but I think it is justified—I can make it more rigorous if need be.
Then you must be making a lot of exceptions, or you don’t downvote very much
Indeed I don’t downvote very much; although probably more than you’re thinking, since I on reflection I don’t typically explain my votes if they don’t affect the sign of the comment’s score.
Could you imagine if every 4th comment in ‘recent comments’ was taken up by my explanations of why I downvoted a comment?
I think you downvote too much. My perception is that, other than the rapid downvoting of trolls and inane comments, the quality of this site is the result mainly of the incentives created by upvoting, rather than downvoting.
Yes, too much explanation would also be bad; but jimrandomh apparently wants none, and I vigorously oppose that. The right to inquire about a downvote should not be trampled upon!
However, it can feel really irritating to get downvoted, especially if one doesn’t know why. It happens to all of us sometimes, and it’s perfectly acceptable to ask for an explanation.
Perhaps we have different ideas of what ‘rights’ and ‘trampling upon’ rights entail.
You have the right to comment about reasons for downvoting—no one will stop you and armed guards will not show up and beat you for it. I think it is a good thing that you have this right.
If I think we would be better off with fewer comments like that, I’m fully within my rights to downvote the comment; similarly, no one will stop me and armed guards will not show up and beat me for it. I think it is a good thing that I have this right.
I’m not sure in what sense you think there is a contradiction between those two things, or if we are just talking past each other.
I think you should be permitted to downvote as you please, but do note that literal armed guards are not necessary for there to be real problems with the protection of rights.
My implicit premise was that 1) violent people or 2) a person actually preventing your action are severally necessary for there to be real problems with the protection of rights. Is there a problem with that version?
In such a context, when someone speaks of the “right” to do X, that means the ability to do X without being punished (in whatever way is being discussed). Here, downvoting is the analogue of armed guards beating one up.
Responding by pointing out that a yet harsher form of punishment is not being imposed is not a legitimate move, IMHO.
You are all talking about this topic, and yet you regard me as weird??? That’s like the extrusion die asserting that the metal wire has a grey spectral component!
In such a context, when someone speaks of the “right” to do X, that means the ability to do X without being punished (in whatever way is being discussed). Here, downvoting is the analogue of armed guards beating one up.
Ah, I could see how you would see that as a contradiction, then.
In that case, for purposes of this discussion, I withdraw my support for your right to do that.
And since I intend to downvote any comment or post for any reason I see fit at the time, it follows that no one has the right to post any comment or post of any sort, by your definition, since they can reasonably expect to be ‘punished’ for it.
For the purposes of other discussions, I do not accept your definition of ‘right’, nor do I accept your framing of a downvote as a ‘punishment’ in the relevant sense. I will continue to do my very best to ensure that only the highest-quality content is shown to new users, and if you consider that ‘punishment’, that is irrelevant to me.
Here, downvoting is the analogue of armed guards beating one up.
Wouldn’t that analogue better apply to publicly and personally insulting the poster, targeting your verbal abuse at the very attributes that this community holds dear, deleting posts and threatening banning? Although I suppose your analogous scale could be extended in scope to include ‘imprisonment and torture without trial’.
On the topic of the immediate context I do hope that you consider thomblakes position and make an exception to your usual policy in his case. I imagine it would be extremely frustrating for you to treat others with what you consider to be respect and courtesy when you know that the recipient does not grant you the same right. It would jar with my preference for symmetry if I thought you didn’t feel free to implement a downvote friendly voting policy at least on a case by case basis. I wouldn’t consider you to be inconsistent, and definitely not hypocritical. I would consider you sane.
The proper reason to request clarification is in order to not make the mistake again—NOT as a defensive measure against some kind of imagined slight on your social status. Yes social status is a part of the reason for the karma system—but it is not something you have an inherent right to. Otherwise there would be no point to it!
Some good reasons to be downvoted: badly formed assertions, ambiguous statements, being confidently wrong, being belligerent, derailing the topic.
In this case your statement was a vague disagreement with the intuitively correct answer, with no supporting argument provided. That is just bad writing, and I would downvote it for so being. It does not imply that I think you have no real idea (something I have no grounds to take a position on), just that the specific comment did not communicate your idea effectively. You should value such feedback, as it will help you improve your writing skills,
The proper reason to request clarification is in order to not make the mistake again
I reject out of hand any proposed rule of propriety that stipulates people must pretend to be naive supplicants.
When people ask me for an explanation of a downvote I most certainly do not take it for granted that by so doing they are entering into my moral reality and willing to accept my interpretation of what is right and what is a ‘mistake’. If I choose to explain reasons for a downvote I also don’t expect them to henceforth conform to my will. They can choose to keep doing whatever annoying thing they were doing (there are plenty more downvotes where that one came from.)
There is more than one reason to ask for clarification for a downvote—even “I’m just kinda curious” is a valid reason. Sometimes votes just seem bizarre and not even Machiavellian reasoning helps explain the pattern. I don’t feel obliged to answer any such request but I do so if convenient. I certainly never begrudge others the opportunity to ask if they do so politely.
Yes social status is a part of the reason for the karma system—but it is not something you have an inherent right to. Otherwise there would be no point to it!
I reject out of hand any proposed rule of propriety that stipulates people must pretend to be naive supplicants.
I never said anything about pretending anything. I said if you request clarification, and don’t actually need clarification, you’re just making noise. Ideally you will be downvoted for that.
There is more than one reason to ask for clarification for a downvote—even “I’m just kinda curious” is a valid reason. Sometimes votes just seem bizarre and not even Machiavellian reasoning helps explain the pattern. I don’t feel obliged to answer any such request but I do so if convenient. I certainly never begrudge others the opportunity to ask if they do so politely.
Sure, but I still maintain that a request for clarification itself can be annoying and hence downvote worthy. I don’t think any comment is inherently protected or should be exempt from being downvoted.
Sure, but I still maintain that a request for clarification itself can be annoying and hence downvote worthy. I don’t think any comment is inherently protected or should be exempt from being downvoted.
I agree with you on these points. I downvote requests for clarification sometime—particularly if, say, the reason for the downvote is transparent or the flow conveys an attitude that jars with me. I certainly agree that people should be free to downvote freely whenever they please and for whatever reason they please—again, for me to presume otherwise would be a demand for naivety or dishonesty (typically both).
Feedback is valuable when it is informative, as the exchange with WrongBot turned out to be in the end.
Unfortunately, a downvote by itself will not typically be that informative. Sometimes it’s obvious why a comment was downvoted (in which case it doesn’t provide much information anyway); but in this case, I had no real idea, and it seemed plausible that it resulted from a misinterpretation of the comment. (As turned out to be the case.)
(Also, the slight to one’s social status represented by a downvote isn’t “imagined”; it’s tangible and numerical.)
In this case your statement was a vague disagreement with the intuitively correct answer, with no supporting argument provided. That is just bad writing, and I would downvote it for so being
The comment was a quick answer to a yes-no question posed to me by Eliezer. Would you have been more or less inclined to downvote it if I had written only “Yes”?
Unfortunately, a downvote by itself will not typically be that informative. Sometimes it’s obvious why a comment was downvoted (in which case it doesn’t provide much information anyway); but in this case, I had no real idea, and it seemed plausible that it resulted from a misinterpretation of the comment. (As turned out to be the case.)
Providing information isn’t the point of downvoting, it is a means of expressing social disapproval. (Perhaps that is information in a sense, but it is more complicated than just that.) The fact that they are being contrary to a social norm may or may not be obvious to the commenter, if not then it is new information. Regardless, the downvote is a signal to reexamine the comment and think about why it was not approved by over 50% of the readers who felt strongly enough to vote on it.
(Also, the slight to one’s social status represented by a downvote isn’t “imagined”; it’s tangible and numerical.)
Tangibility and significance are completely different matters. A penny might appear more solid than a dollar, but is far less worthy of consideration. You could ignore a minus-1 comment quite safely without people deciding (even momentarily) that you are a loser or some such. That you chose not to makes it look like you have an inflated view of how significant it is.
The comment was a quick answer to a yes-no question posed to me by Eliezer. Would you have been more or less inclined to downvote it if I had written only “Yes”?
Probably less, as I would then have simply felt like requesting clarification, or perhaps even thinking of a reason on my own. A bad argument (or one that sounds bad) is worse than no argument.
Status matters; it’s a basic human desideratum, like food and sex (in addition to being instrumentally useful in various ways). There seems to be a notion among some around here that concern with status is itself inherently irrational or bad in some way. But this is as wrong as saying that concern with money or good-tasting food is inherently irrational or bad. Yes, we don’t want the pursuit of status to interfere with our truth-detecting abilities; but the same goes for the pursuit of food, money, or sex, and no one thinks it’s wrong for aspiring rationalists to pursue those things.
Status is an inherently zero-sum good, so while it is rational for any given individual to pursue it; we’d all be better off, cet par, if nobody pursued it. Everyone has a small incentive for other people not to pursue status, just as they have an incentive for them not to be violent or to smell funny; hence the existence of popular anti-status-seeking norms.
I don’t think I agree, at least in the present context. I think of status as being like money—or, in fact, the karma score on LW, since that is effectively what we’re talking about here anyway. It controls the granting of important privileges, such as what we might call “being listened to”—having folks read your words carefully, interpret them charitably, and perhaps even act on them or otherwise be influenced by them.
(To tie this to the larger context, this is why I started paying attention to SIAI: because Eliezer had won “status” in my mind.)
While status may appear zero-sum amongst those who are competing for influence in a community, for the community as a whole, status is postive sum when in it accurately reflects the value of people to the community.
I don’t think I agree, at least in the present context. I think of status as being like money—or, in fact, the karma score on LW, since that is effectively what we’re talking about here anyway. It controls the granting of important privileges, such as what we might call “being listened to”—having folks read your words carefully, interpret them charitably, and perhaps even act on them or otherwise be influenced by them.
(To tie this to the larger context, this is why I started paying attention to SIAI: because Eliezer had won “status” in my mind.)
This view arises from what I understand about the “modular” nature of the human brain: we think we’re a single entity that is “flexible enough” to think about lots of different things, but in reality our brains consist of a whole bunch of highly specialized “modules”, each able to do some single specific thing.
The brain has many different components with specializations, but the largest and human dominant portion, the cortex, is not really specialized at all in the way you outline.
The cortex is no more specialized than your hard drive.
Its composed of a single repeating structure and associated learning algorithm that appears to be universal. The functional specializations that appear in the adult brain arise due to topological wiring proximity to the relevant sensory and motor connections. The V1 region is not hard-wired to perform mathematically optimal gabor-like edge filters. It automatically evolves into this configuration because it is the optimal configuration for modelling the input data at that layer, and it evolves thus soley based on exposure to said input data from retinal ganglion cells.
You can think of cortical tissue as a biological ‘neuronium’. It has a semi-magical emergent capacity to self-organize into an appropriate set of feature detectors based on what its wired to. more on this
All that being said, the inter-regional wiring itself is currently less understood and is probably more genetically predetermined.
Well, it seems we disagree. Honestly, I see the problem of AGI as the fairly concrete one of assembling an appropriate collection of thousands-to-millions of “narrow AI” subcomponents.
There may be other approaches that are significantly simpler (that we haven’t yet found, obviously). Assuming AGI happens, it will have been a race between the specific (type of) path you imagine, and every other alternative you didn’t think of. In other words, you think you have an upper bound on how much time/expense it will take.
I’m not a member of SIAI but my reason for thinking that AGI is not just going to be like lots of narrow bits of AI stuck together is that I can see interesting systems that haven’t been fully explored (due to difficulty of exploration). These types of systems might solve some of the open problems not addressed by narrow AI.
These are problems such as
How can a system become good at so many different things when it starts off the same. Especially puzzling is how people build complex (unconscious) machinery for dealing with problems that we are not adapted for, like Chess.
How can a system look after/upgrade itself without getting completely pwned by malware (We do get partially pwned by hostile memes, but is not complete take over of the same type as getting rooted).
Now I also doubt that these systems will develop quickly when people get around to investigating them. And they will have elements of traditional narrow AI in as well, but they will be changeable/adaptable parts of the system, not fixed sub-components. What I think needs is exploring is primarily changes in software life-cycles rather than a change in the nature of the software itself.
And Learning is equivalent to absorbing memes. The two are one and the same.
I don’t agree. Meme absorption is just one element of learning.
To learn how to play darts well you absorb a couple of dozen memes and then spend hours upon hours rewiring your brain to implement a complex coordination process.
To learn how to behave appropriately in a given culture you learn a huge swath of existing memes, continue to learn a stream of new ones but also dedicate huge amounts of background processing reconfiguring the weightings of existing memes relative to each other and external inputs. You also learn all sorts of implicit information about how memes work for you specifically (due to, for example, physical characteristics), much of this information will never be represented in meme form.
Fine, if you take memes to be just symbolic level transferable knowledge (which, thinking it over, I agree with), then at a more detailed level learning involves several sub-processes, one of which is the rapid transfer of memes into short term memory.
I don’t think AGI in a few decades is very farfetched at all. There’s a heckuvalot of neuroscience being done right now (the Society for Neuroscience has 40,000 members), and while it’s probably true that much of that research is concerned most directly with mere biological “implementation details” and not with “underlying algorithms” of intelligence, it is difficult for me to imagine that there will still be no significant insights into the AGI problem after 3 or 4 more decades of this amount of neuroscience research.
Of course there will be significant insights into the AGI problem over the coming decades—probably many of them. My point was that I don’t see AGI as hard because of a lack of insights; I see it as hard because it will require vast amounts of “ordinary” intellectual labor.
I’m having trouble understanding how exactly you think the AGI problem is different from any really hard math problem. Take P != NP, for instance the attempted proof that’s been making the rounds on various blogs. If you’ve skimmed any of the discussion you can see that even this attempted proof piggybacks on “vast amounts of ‘ordinary’ intellectual labor,” largely consisting of mapping out various complexity classes and their properties and relations. There’s probably been at least 30 years of complexity theory research required to make that proof attempt even possible.
I think you might be able to argue that even if we had an excellent theoretical model of an AGI, that the engineering effort required to actually implement it might be substantial and require several decades of work (e.g. Von Neumann architecture isn’t suitable for AGI implementation, so a great deal of computer engineering has to be done).
If this is your position, I think you might have a point, but I still don’t see how the effort is going to take 1 or 2 centuries. A century is a loooong time. A century ago humans barely had powered flight.
Take P != NP, for instance the attempted proof that’s been making the rounds on various blogs. If you’ve skimmed any of the discussion you can see that even this attempted proof piggybacks on “vast amounts of ‘ordinary’ intellectual labor,
By no means do I want to downplay the difficulty of P vs NP; all the same, I think we have different meanings of “vast” in mind.
The way I think about it is: think of all the intermediate levels of technological development that exist between what we have now and outright Singularity. I would only be half-joking if I said that we ought to have flying cars before we have AGI. There are of course more important examples of technologies that seem easier than AGI, but which themselves seem decades away. Repair of spinal cord injuries; artificial vision; useful quantum computers (or an understanding of their impossibility); cures for the numerous cancers; revival of cryonics patients; weather control. (Some of these, such as vision, are arguably sub-problems of AGI: problems that would have to be solved in the course of solving AGI.)
Actually, think of math problems if you like. Surely there are conjectures in existence now—probably some of them already famous—that will take mathematicians more than a century from now to prove (assuming no Singularity or intelligence enhancement before then). Is AGI significantly easier than the hardest math problems around now? This isn’t my impression—indeed, it looks to me more analogous to problems that are considered “hopeless”, like the “problem” of classifying all groups, say.
By no means do I want to downplay the difficulty of P vs NP; all the same, I think we have different meanings of “vast” in mind.
I hate to go all existence proofy on you, but we have an existence proof of a general intelligence—accidentally sneezed out by natural selection, no less, which has severe trouble building freely rotating wheels—and no existence proof of a proof of P != NP. I don’t know much about the field, but from what I’ve heard, I wouldn’t be too surprised if proving P != NP is harder than building FAI for the unaided human mind. I wonder if Scott Aaronson would agree with me on that, even though neither of us understand the other’s field? (I just wrote him an email and asked, actually; and this time remembered not to say my opinion before asking for his.)
After glancing over a 100-page proof that claimed to solve the biggest problem in computer science, Scott Aaronson bet his house that it was wrong. Why?
What I find interesting is that the pattern nearly always goes the other way: you’re more likely to think that a celebrated problem you understand well is harder than one you don’t know much about. It says a lot about both Eliezer’s and Scott’s rationality that they think of the other guy’s hard problems as even harder than their own.
As for existence proof of a general intelligence, that doesn’t prove anything about how difficult it is, for anthropic reasons. For all we know 10^20 evolutions each in 10^50 universes that would in principle allow intelligent life might on average result in 1 general intelligence actually evolving.
Of course, if you buy the self-indication assumption (which I do not) or various other related principles you’ll get an update that compels belief in quite frequent life (constrained by the Fermi paradox and a few other things).
More relevantly, approaches like Robin’s Hard Step analysis and convergent evolution (e.g. octopus/bird intelligence) can rule out substantial portions of “crazy-hard evolution of intelligence” hypothesis-space. And we know that human intelligence isn’t so unstable as to see it being regularly lost in isolated populations, as we might expect given ludicrous anthropic selection effects.
We can make better guesses than that: evolution coughed up quite a few things that would be considered pretty damn intelligent for a computer program, like ravens, octopuses, rats or dolphins.
Not independently (not even cephalopods, at least completely). And we have no way of estimating the difference in difficulty between that level of intelligence and general intelligence other than evolutionary history (which for anthropic reasons could be highly untypical), and similarity in makeup, but already know that our type of nervous system is capable of supporting general intelligence, most rat level intelligences might hit fundamental architectural problems first.
We can always estimate, even with very little knowledge—we’ll just have huge error margins. I agree it is possible that “For all we know 10^20 evolutions each in 10^50 universes that would in principle allow intelligent life might on average result in 1 general intelligence actually evolving”, I would just bet on a much higher probability than that, though I agree with the principle.
The evidence that pretty smart animals exist in distant branches of the tree of life, and in different environments is weak evidence that intelligence is “pretty accessible” in evolution’s search space. It’s stronger evidence than the mere fact that we, intelligent beings, exist.
Intelligence sure. The original point was that our existence doesn’t put a meaningful upper bound on the difficultly of general intelligence. Cephalopods are good evidence that given whatever rudimentary precursors of a nervous system our common ancestor had (I know it had differentiated cells, but I’m not sure what else. I think it didn’t really have organs like higher animals, let alone anything that really qualified as a nervous system) cephalopod level intelligence is comparatively easy, having evolved independently two times. It doesn’t say anything about how much more difficult general intelligence is compared to cephalopod intelligence, nor whether whatever precursors to a nervous system our common ancestor had were unusually conductive to intelligence compared to the average of similar complex evolved beings.
If I had to guess I would assume cephalopod level intelligence within our galaxy and a number of general intelligences somewhere outside our past light cone. But that’s because I already think of general intelligence as not fantastically difficult independently of the relevance of the existence proof.
Hox genes suggest that they both had a modular body plan of some sort. Triploblasty implies some complexity (the least complex triploblastic organism today is a flatworm).
I’d be very surprised if most recent common ancestor didn’t have neurons similar to most neurons today, as I’ve had a hard time finding out the differences between the two. A basic introduction to nervous systems suggests they are very similar.
Well, I for one strongly hope that we resolve whether P = NP before we have AI since a large part of my estimate for the probability of AI being able to go FOOM is based on how much of the complexity hierarchy collapses. If there’s heavy collapse, AI going FOOM Is much more plausible.
I don’t know much about the field, but from what I’ve heard, I wouldn’t be too surprised if proving P != NP is harder than building FAI for the unaided human mind
Well actually, after thinking about it, I’m not sure I would either. There is something special about P vs NP, from what I understand, and I didn’t even mean to imply otherwise above; I was only disputing the idea that “vast amounts” of work had already gone into the problem, for my definition of “vast”.
Scott Aaronson’s view on this doesn’t move my opinion much (despite his large contribution to my beliefs about P vs NP), since I think he overestimates the difficulty of AGI (see your Bloggingheads diavlog with him).
I don’t know much about the field, but from what I’ve heard, I wouldn’t be too surprised if proving P != NP is harder than building FAI for the unaided human mind.
Awesome! Be sure to let us know what he thinks. Sounds unbelievable to me though, but what do I know.
A ‘few clues’ sounds like a gross underestimation. It is the only working example, so it certainly contains all the clues, not just a few. The question of course is how much of a shortcut is possible. The answer to date seems to be: none to slim.
I agree engineers reverse engineering will succeed way ahead of full emulation, that wasn’t my point.
If information is not extracted and used, it doesn’t qualify as being a “clue”.
The question of course is how much of a shortcut is possible.
The answer to date seems to be: none to slim.
The search oracles and stockmarketbot makers have paid precious little attention to the brain. They are based on engineering principles instead.
I agree engineers reverse engineering will succeed way ahead of full emulation,
Most engineers spend very little time on reverse-engineering nature. There is a little “bioinspiration”—but inspiration is a bit different from wholescale copying.
but I still don’t see how the effort is going to take 1 or 2 centuries. A century is a loooong time.
I think the following quote is illustrative of the problems facing the field:
After [David Marr] joined us, our team became the most famous vision group in the world, but the one with the fewest results. His idea was a disaster. The edge finders they have now using his theories, as far as I can see, are slightly worse than the ones we had just before taking him on. We’ve lost twenty years.
-Marvin Minsky, quoted in “AI” by Daniel Crevier.
Some notes and interpretation of this comment:
Most vision researchers, if asked who is the most important contributor to their field, would probably answer “David Marr”. He set the direction for subsequent research in the field; students in introductory vision classes read his papers first.
Edge detection is a tiny part of vision, and vision is a tiny part of intelligence, but at least in Minsky’s view, no progress (or reverse progress) was achieved in twenty years of research by the leading lights of the field.
There is no standard method for evaluating edge detector algorithms, so it is essentially impossible to measure progress in any rigorous way.
I think this kind of observation justifies AI-timeframes on the order of centuries.
Edge detection is rather trivial. Visual recognition however is not, and there certainly are benchmarks and comparable results in that field. Have you browsed the recent pubs of Poggio et al at MIT vision lab? There is lots of recent progress, with results matching human levels for quick recognition tasks.
Also, vision is not a tiny part of intelligence. Its the single largest functional component of the cortex, by far. The cortex uses the same essential low-level optimization algorithm everywhere, so understanding vision at the detailed level is a good step towards understanding the whole thing.
And finally and most relevant for AGI, the higher visual regions also give us the capacity for visualization and are critical for higher creative intelligence. Literally all scientific discovery and progress depends on this system.
“visualization is the key to enlightenment” and all that
It’s only trivial if you define an “edge” in a trivial way, e.g. as a set of points where the intensity gradient is greater than a certain threshold. This kind of definition has little use: given a picture of a tree trunk, this definition will indicate many edges corresponding to the ridges and corrugations of the bark, and will not highlight the meaningful edge between the trunk and the background.
I don’t believe that there is much real progress recently in vision. I think the state of the art is well illustrated by the “racist” HP web camera that detects white faces but not black faces.
Also, vision is not a tiny part of intelligence [...] The cortex uses the same essential low-level optimization algorithm everywhere,
I actually agree with you about this, but I think most people on LW would disagree.
Whether you are talking about canny edge filters, gabor like edge detection more similar to what V1 self-organizes into, they are all still relatively simple—trivial compared to AGI. Trivial as in something you code in a few hours for your screen filter system in a modern game render engine.
The particular problem you point out with the tree trunk is a scale problem and is easily handled in any good vision system.
An edge detection filter is just a building block, its not the complete system.
In HVS, initial edge preprocessing is done in the retina itself which essentially does on-center, off-surround gaussian filters (similar to low-pass filters in photoshop). The output of the retina is thus essentially a multi-resolution image set, similar to a wavelet decomposition. The image output at this stage becomes a series of edge differences (local gradients), but at numerous spatial scales.
The high frequency edges such as the ridges and corrugations of the bark are cleanly separated from the more important low frequency edges separating the tree trunk from the background. V1 then detects edge orientations at these various scales, and higher layers start recognizing increasingly complex statistical patterns of edges across larger fields of view.
Whether there is much real progress recently in computer vision is relative to one’s expectations, but the current state of the art in research systems at least is far beyond your simplistic assessment. I have a layman’s overview of HVS here. If you really want to know about the current state of the art in research, read some recent papers from a place like Poggio’s lab at MIT.
In the product space, the HP web camera example is also very far from the state of the art, I’m surprised that you posted that.
There is free eye tracking software you can get (running on your PC) that can use your web cam to track where your eyes are currently focused in real time. That’s still not even the state of the art in the product space—that would probably be the systems used in the more expensive robots, and of course that lags the research state of the art.
AIXI’s contribution is more philosophical than practical. I find a depressing over-emphasis of bayesian probability theory here as the ‘math’ of choice vs computational complexity theory, which is the proper domain.
The most likely outcome of a math breakthrough will be some rough lower and or upper bounds on the shape of the intelligence over space/time complexity function. And right now the most likely bet seems to be that the brain is pretty well optimized at the circuit level, and that the best we can do is reverse engineer it.
EY and the math folk here reach a very different conclusion, but I have yet to find his well considered justification. I suspect that the major reason the mainstream AI community doesn’t subscribe to SIAI’s math magic bullet theory is that they hold the same position outline above: ie that when we get the math theorems, all they will show is what we already suspect: human level intelligence requires X memory bits and Y bit ops/second, where X and Y are roughly close to brain levels.
This, if true, kills the entirety of the software recursive self-improvement theory. The best that software can do is approach the theoretical optimum complexity class for the problem, and then after that point all one can do is fix it into hardware for a further large constant gain.
right now the most likely bet seems to be that the brain is pretty well optimized
at the circuit level, and that the best we can do is reverse engineer it.
That seems like crazy talk to me. The brain is not optimal—not its hardware or software—and not by a looooong way! Computers have already steam-rollered its memory and arithmetic -units—and that happened before we even had nanotechonolgy computing components. The rest of the brain seems likely to follow.
Edit: removed a faulty argument at the end pointed out by wedrifid.
I am talking about optimality for AGI in particular with respect to circuit complexity, with the typical assumptions that a synapse is vaguely equivalent to a transistor, maybe ten transistors at most. If you compare on that level, the brain looks extremely efficient given how slow the neurons are. Does this make sense?
The brain’s circuits have around 10^15 transistor equivalents, and a speed of 10^3 cycles per second. 10^18 transistor cycles / second
A typical modern CPU has 10^9 transistors, with a speed of 10^9 cycles per second. 10^18 transistor cycles / second
Our CPU’s strength is not their circuit architecture or software—its the raw speed of CMOS, its a million X substrate advantage. The learning algorithm, the way in which the cortex rewires in response to input data, appears to be a pretty effective universal learning algorithm.
The brain’s architecture is a joke. It is as though a telecoms engineer decided to connect a whole city’s worth of people together by running cables directly between any two people who wanted to have a chat. It hasn’t even gone fully digital yet—so things can’t easily be copied or backed up. The brain is just awful—no wonder human cognition is such a mess.
Then some questions: How long would moore’s law have to continue into the future with no success in AGI for that to show that the brain’s is well optimized for AGI at the circuit level?
I’ve taken some attempts to show rough bounds on the brain’s efficiency, are you aware of some other approach or estimate?
Then some questions: How long would moore’s law have to continue
into the future with no success in AGI for that to show that the brain’s
is well optimized for AGI at the circuit level?
Most seem to think the problem is mostly down to software—and that supercomputer hardware is enough today—in which case more hardware would not necessarily help very much. The success or failure of adding more hardware might give an indication of how hard it is to find the target of intelligence in the search space. It would not throw much light on the issue of how optimally “designed” the brain is. So: your question is a curious one.
The success or failure of adding more hardware might give an indication of how hard it is to find the target of intelligence in the search space
For every computational system and algorithm, there is a minimum level of space-time complexity in which this system can be encoded. As of yet we don’t know how close the brain is to the minimum space-time complexity design for an intelligence of similar capability.
Lets make the question more specific: whats the minimum bit representation of a human-equivalent mind? If you think the brain is far off that, how do you justify that?
Of course more hardware helps: it allows you to search through the phase space faster. Keep in mind the enormity of the training time.
I happen to believe the problem is ‘mostly down to software’, but I don’t see that as a majority view—the Moravec/Kurzweil view that we need brain-level hardware (within an order of magnitude or so) seems to be majoritive at this point.
We need brain-level hardware (within an order of magnitude or so) if machines are going to be cost-competitive with humans. If you just want a supercomputer mind, then no problem.
I don’t think Moravec or Kurzweil ever claimed it was mostly down to hardware. Moravec’s charts are of hardware capability—but that was mainly because you can easily measure that.
We need brain-level hardware (within an order of magnitude or so) if machines are going to be cost-competitive with humans.
I don’t see why that is. If you were talking about ems, then the threshhold should be 1:1 realtime. Otherwise, for most problems that we know how to program a computer to do, the computer is much faster than humans even at existing speeds. Why do you expect that a computer that’s say, 3x slower than a human (well within an order of magnitude) would be cost-competitive with humans while one that’s 10^4 times slower wouldn’t?
Evidently there are domains where computers beat humans today—but if you look at what has to happen for machines to take the jobs of most human workers, they will need bigger and cheaper brains to do that. “Within an order of magnitude or so” seems like a reasonable ballpark figure to me. If you are looking for more details about why I think that, they are not available at this time.
I suspect that the controlling reason why you think that is that you assume it takes human-like hardware to accomplish human-like tasks, and greatly underestimate the advantages of a mind being designed rather than evolved.
Lets make the question more specific: whats the minimum bit representation of a human-equivalent mind?
Way off. Let’s see… I would bet at even odds that it is 4 or more orders of magnitude off optimal.
If you think the brain is far off that, how do you justify that?
We have approximately one hundred billion neurons each and roughly the same number of glial cells (more of the latter if we are smart!). Each of those includes a full copy of our DNA, which is itself not exactly optimally compressed.
Way off. Let’s see… I would bet at even odds that it is 4 or more orders of magnitude off optimal.
you didn’t answer my question: what is your guess at minimum bit representation of a human equi mind?
you didn’t use the typical methodology of measuring the brain’s storage, nor did you provide another.
I wasn’t talking about molecular level optimization. I started with the typical assumption that synapses represent a few bits, the human brain has around 100TB to 1PB of data/circuitry, etc etc—see the singularity is near.
So you say the human brain algorithmic representation is off by 4 orders of magnitude or more—you are saying that you think a human equivalent mind can be represented in 10 to 100GB of data/circuitry?
If so, why did evolution not find that by now? It has had plenty of time to compress at the circuit level. In fact, we actually know that the brain does perform provably optimal compression on its input data in a couple of domains—see V1 and its evolution into gabor-like edge feature detection.
Evolution has had plenty of time to find a well-optimized cellular machinery based on DNA, plenty of time to find a well-optimized electro-chemical computing machinery based on top of that, and plenty of time to find well-optimized circuits within that space.
Even insects are extremely well-optimized at the circuit level—given their neuron/synapse counts, we have no evidence whatsoever to believe that vastly simpler circuits exist that can perform the same functionality.
When we have used evolutionary exploration algorithms to design circuits natively, given enough time we see similar complex, messy, but near optimal designs, and this is a general trend.
Are you saying that you are counting every copy of the DNA as information that contributes to the total amount? If so, I say that’s invalid. What if each cell were remotely controlled from a central server containing the DNA information? I can’t see that we’d count the DNA for each cell then—yet it is no different really.
I agree that the number of cells is relevant, because there will be a lot of information in the structure of an adult brain that has come from the environment, rather than just from the DNA, and more cells would seem to imply more machinery in which to put it.
Are you saying that you are counting every copy of the DNA as information that contributes to the total amount? If so, I say that’s invalid. What if each cell were remotely controlled from a central server containing the DNA information? I can’t see that we’d count the DNA for each cell then—yet it is no different really.
I thought we were talking about the efficiency of the human brain. Wasn’t that the whole point? If every cell is remotely controlled from a central server then well, that’d be whole different algorithm. In fact, we could probably scrap the brain and just run the central server.
Genes actually do matter in the functioning of neurons. Chemical additions (eg. ethanol) and changes in the environment (eg. hypoxia) can influence gene expression in cells in the brain, impacting on their function.
I suggest the brain is a ridiculously inefficient contraption thrown together by the building blocks that were practical for production from DNA representations and suitable for the kind of environments animals tended to be exposed to. We should be shocked to find that it also manages to be anywhere near optimal for general intelligence. Among other things it would suggest that evolution packed the wrong lunch.
Okay, I may have misunderstood you. It looks like there is some common ground between us on the issue of inefficiency. I think the brain would probably be inefficient as well as it has to be thrown together by the very specific kind of process of evolution—which is optimized for building things without needing look-ahead intelligence rather than achieving the most efficient results.
Then some questions: How long would moore’s law have to continue into the future with no success in AGI for that to show that the brain’s is well optimized for AGI at the circuit level?
A Sperm Whale and a bowl of Petunias.
My first impulse was to answer that Moore’s law could go forever and never produce success in AGI, since ‘AGI’ isn’t just what you get when you put enough computronium together for it to reach critical mass. But even given no improvements in understanding we could very well arrive at AGI just through ridiculous amounts of brute force. In fact, given enough space and time, randomised initial positions and possibly a steady introduction of negentropy we could produce an AGI in Conways Life.
I’ve taken some attempts to show rough bounds on the brain’s efficiency, are you aware of some other approach or estimate?
You could find some rough bounds by seeing how many parts of a human brain you can cut out without changing IQ.Trivial little things like, you know, the pre-frontal cortex.
You are just talking around my questions, so let me make it more concrete. An important task of any AGI is higher level sensor data interpretation—ie seeing. We have an example system in the human brain—the human visual system, which is currently leaps and bounds beyond the state of the art in machine vision. (although the latter is making progress towards the former through reverse engineering)
So machine vision is a subtask of AGI. What is the minimal computational complexity of human-level vision? This is a concrete computer science problem. It has a concrete answer—not “sperm whale and petunia” nonsense.
Until someone makes a system better than HVS, or proves some complexity bounds, we don’t know how optimal HVS is for this problem, but we also have no reason to believe that it is orders of magnitude off from the theoretic optimum.
Good quality general-purpose data-compression would “break the back” of the task of buliding synthetic intelligent agents—and that’s a “simple” math problem—as I explain on: http://timtyler.org/sequence_prediction/
At least it can be stated very concisely. Solutions so far haven’t been very simple—but the brain’s architecture offers considerable hope for a relatively simple solution.
Note that allowing for a possibility of sudden breakthrough is also an antiprediction, not a claim for a particular way things are. You can’t know that no such thing is possible, without having understanding of the solution already at hand, hence you must accept the risk. It’s also possible that it’ll take a long time.
I’m reading through and catching up on this thread, and rather strongly agreed with your statement:
Eliezer and others at SIAI to assign (relatively) large amounts of probability mass to the scenario of a small set of people having some “insight” which allows them to suddenly invent AGI in a basement. In other words, they tend to view AGI as something like an unsolved math problem, like those on the Clay Millennium list, whereas it seems to me like a daunting engineering task analogous to colonizing Mars (or maybe Pluto).
However, pondering it again, I realize there is an epistemological spectrum ranging from math on the one side to engineering on the other. Key insights into new algorithms can undoubtedly speed up progress, and such new insights often can be expressed as pure math, but at the end of the day it is a grand engineering (or reverse engineering) challenge.
However, I’m somewhat taken aback when you say, “the notion that AGI is only decades away, as opposed to a century or two.”
In other words, I largely agree with Ben Goertzel’s assertion that there is a fundamental difference between “narrow AI” and AI research that might eventually lead to machines capable of cognition, but I’m not sure I have good evidence for this argument.
One obvious piece of evidence is that many forms of narrow learning are mathematically incapable of doing much. There are for example a whole host of theorems about what different classes of neural networks can actually recognize, and the results aren’t very impressive. Similarly, support vector machine’s have a lot of trouble learning anything that isn’t a very simple statistical model, and even then humans need to decide which stats are relevant. Other linear classifiers run into similar problems.
I work in this field, and was under approximately the opposite impression; that voice and visual recognition are rapidly approaching human levels. If I’m wrong and there are sharp limits, I’d like to know. Thanks!
Machine intelligence has surpassed “human level” in a number of narrow domains. Already, humans can’t manipulate enough data to do anything remotely like a search engine or a stockbot can do.
The claim seems to be that in narrow domains there are often domain-specific “tricks”—that wind up not having much to do with general intelligence—e.g. see chess and go. This seems true—but narrow projects often broaden out. Search engines and stockbots really need to read and understand the web. The pressure to develop general intelligence in those domains seems pretty strong.
Those who make a big deal about the distinction between their projects and “mere” expert systems are probably mostly trying to market their projects before they are really experts at anything.
One of my videos discusses the issue of whether the path to superintelligent machines will be “broad” or “narrow”:
Thanks, it always is good to actually have input from people who work in a given field. So please correct me if I’m wrong but I’m under the impression that
1) neutral networks cannot in general detect connected components unless the network has some form of recursion.
2) No one knows how to make a neural network with recursion learn in any effective, marginally predictable fashion.
This is the sort of thing I was thinking of. Am I wrong about 1 or 2?
Not sure what you mean about by 1), but certainly, recurrent neural nets are more powerful. 2) is no longer true; see for example the GeneRec algorithm. It does something much like backpropagation, but with no derivatives explicitly calculated, there’s no concern with recurrent loops.
On the whole, neural net research has slowed dramatically based on the common view you’ve expressed; but progress continues apace, and they are not far behind cutting edge vision and speech processing algorithms, while working much more like the brain does.
Thanks. GeneRec sounds very interesting. Will take a look. Regarding 1, I was thinking of something like the theorems in chapter 9 in Perceptrons which shows that there are strong limits on what topological features of input a non-recursive neural net can recognize.
I think multifoliaterose is right that there’s a PR problem, but it’s not just a PR problem. It seems, unfortunately, to be a problem with having enough justification for claims, and a problem with connecting to the world of professional science. I think the PR problems arise from being too disconnected from the demands placed on other scientific or science policy organizations. People who study other risks, say epidemic disease, have to get peer-reviewed, they have to get government funding—their ideas need to pass a round of rigorous criticism. Their PR is better by necessity.
I agree completely. The reason why I framed my top level post in the way that I did was so that it would be relevant to readers of a variety of levels of confidence in SIAI’s claims.
As I indicate here, I personally wouldn’t be interested in funding SIAI as presently constituted even if there was no PR problem.
First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies. (This is also the basis of my skepticism about cryonics.) If you’re going to say “Technology X is likely to be developed” then I’d like to see your prediction mechanism and whether it’s worked in the past.
I think there are ways to make these predictions. On the most layman level I would point out that IBM build a robot that beats people at Jeopardy. Yes, I am aware that this is a complete machine-learning hack (this is what I could gather from the NYT coverage) and is not true cognition, but it surprised even me (I do know something about ML). I think this is useful to defeat the intuition of “machines cannot do that”. If you are truly interested I think you can (I know you’re capable) read Norvig’s AI book, and than follow up on the parts of it that most resemble human cognition; I think serious progress is made in those areas. BTW, Norvig does take FAI issues seriously, including a reference to EY paper in the book.
Second, shouldn’t an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there’s room for pure philosophy and mathematics, but you’d need some grounding in actual AI to understand what future AIs are likely to do.
I think they should, I have no idea if this is being done; but if I would do it I would not do it publicly, as it may have very counterproductive consequences. So until you or I become SIAI fellows we will not know, and I cannot hold such lack of knowledge against them.
First, I’m not really claiming “machines cannot do that.” I can see advances in machine learning and I can imagine the next round of advances being pretty exciting. But I’m thinking in terms of maybe someday a machine being able to distinguish foreground from background, or understand a sentence in English, not being a superintelligence that controls Earth’s destiny. The scales are completely different. One scale is reasonable; one strains credibility, I’m afraid.
Thanks for the book recommendation; I’ll be sure to check it out.
I think controlling Earth’s destiny is only modestly harder than understanding a sentence in English—in the same sense that I think Einstein was only modestly smarter than George W. Bush. EY makes a similar point.
You sound to me like someone saying, sixty years ago: “Maybe some day a computer will be able to play a legal game of chess—but simultaneously defeating multiple grandmasters, that strains credibility, I’m afraid.” But it only took a few decades to get from point A to point B. I doubt that going from “understanding English” to “controlling the Earth” will take that long.
There’s a problem with it, though. Some decades ago you’d have just as eagerly subscribed to this statement: “Controlling Earth’s destiny is only modestly harder than playing a good game of chess”, which we now know to be almost certainly false.
I agree with Rain. Understanding implies a much deeper model than playing. To make the comparison to chess, you would have to change it to something like, “Controlling Earth’s destiny is only modestly harder than making something that can learn chess, or any other board game, without that game’s mechanics (or any mapping from the computer’s output to game moves) being hard-coded, and then play it at an expert level.”
It’s the word “understanding” in the quote which makes it presume general intelligence and/or consciousness without directly stating it. The word “playing” does not have such a connotation, at least to me. I don’t know if I would think differently back when chess required intelligence.
shouldn’t an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there’s room for pure philosophy and mathematics, but you’d need some grounding in actual AI to understand what future AIs are likely to do.
Yes. It’s hardly urgent, since AI researchers are nowhere near a runaway intelligence. But on the other hand, control of AI is going to be crucial+difficult eventually, and it would be good for researchers to be aware of it, if they aren’t.
Right, it’s just (in my and most other AI researchers’[*] opinion) overwhelmingly likely that we are in fact nowhere near (the capability of) it. Although it’s interesting to me that I don’t feel there’s that much difference in probability of “(good enough to) run away improving itself quickly past human level AI” in the next year, and in the next 10 years—both extremely close to 0 is the most specific I can be at this point. That suggests I haven’t really quantified my beliefs exactly yet.
[*] I actually only work on natural language processing using really dumb machine learning, i.e. not general AI.
I am one of those who haven’t been convinced by the SIAI line. I have two main objections.
First, EY is concerned about risks due to technologies that have not yet been developed; as far as I know, there is no reliable way to make predictions about the likelihood of the development of new technologies. (This is also the basis of my skepticism about cryonics.) If you’re going to say “Technology X is likely to be developed” then I’d like to see your prediction mechanism and whether it’s worked in the past.
Second, shouldn’t an organization worried about the dangers of AI be very closely in touch with AI researchers in computer science departments? Sure, there’s room for pure philosophy and mathematics, but you’d need some grounding in actual AI to understand what future AIs are likely to do.
I think multifoliaterose is right that there’s a PR problem, but it’s not just a PR problem. It seems, unfortunately, to be a problem with having enough justification for claims, and a problem with connecting to the world of professional science. I think the PR problems arise from being too disconnected from the demands placed on other scientific or science policy organizations. People who study other risks, say epidemic disease, have to get peer-reviewed, they have to get government funding—their ideas need to pass a round of rigorous criticism. Their PR is better by necessity.
As was mentioned in other threads, SIAI’s main arguments rely on disjunctions and antipredictions more than conjunctions and predictions. That is, if several technology scenarios lead to the same broad outcome, that’s a much stronger claim than one very detailed scenario.
For instance, the claim that AI presents a special category of existential risk is supported by such a disjunction. There are several technologies today which we know would be very dangerous with the right clever ‘recipe’– we can make simple molecular nanotech machines, we can engineer custom viruses, we can hack into some very sensitive or essential computer systems, etc. What these all imply is that a much smarter agent with a lot of computing power is a severe existential threat if it chooses to be.
There needs to be an article on this point. In the absence of a really good way of deciding what technologies are likely to be developed, you are still making a decision. You haven’t signed up yet; whether you like it or not, that is a decision. And it’s a decision that only makes sense if you think technology X is unlikely to be developed, so I’d like to see your prediction mechanism and whether it’s worked in the past. In the absence of really good information, we sometimes have to decide on the information we have.
EDIT: I was thinking about cryonics when I wrote this, though the argument generalizes.
My point, with this, is that everybody is risk-averse and everybody has a time preference. The less is known about the prospects of a future technology, the less willing people are to invest resources into ventures that depend on the future development of that technology. (Whether to take advantage of the technology—as in cryonics—or to mitigate its dangers—as in FAI.) Also, the farther in the future the technology is, the less people care about it; we’re not willing to spend much to achieve benefits or forestall risks in the far future.
I don’t think it’s reasonable to expect people to change these ordinary features of economic preference. If you’re going to ask people to chip in to your cause, and the time horizon is too far, or the uncertainty too high, they’re not going to want to spend their resources that way. And they’ll be justified.
Note: yes, there ought to be some magnitude of benefit or cost that overcomes both risk aversion and time preference. Maybe you’re going to argue that existential risk and cryonics are issues of such great magnitude that they outweigh both risk aversion and time preference.
But: first of all, the importance of the benefit or cost is also an unknown (and indeed subjective.) How much do you value being alive? And, second of all, nobody says our risk and time preferences are well-behaved. There may be a date so far in the future that I don’t care about anything that happens then, no matter how good or how bad. There may be loss aversion—an amount of money that I’m not willing to risk losing, no matter how good the upside. I’ve seen some experimental evidence that this is common.
From what I understand this applies to most people but not everyone, especially outside of contrived laboratory circumstances. Overconfidence and ambition essentially amount to risk-loving choices for some major life choices.
What is it that is making you think that whatever SarahC hasn’t “signed up” to is having a positive effect—and that she can’t do something better with her resources?
Let’s keep in mind that your estimated probabilities of various technological advancements occurring and your level of confidence in those estimates are completely distinct… In particular, here you seem to express low estimated probabilities of various advancements occurring, and you justify this by saying “we really have no idea”. This seems like a complete non sequitur. Maybe you have a correct argument in your mind, but you’re not giving us all the pieces.
Technology X is likely to be developed in a few decades.
Technology X is risky.
We must take steps to mitigate the risk.
If you haven’t demonstrated 1 -- if it’s still unknown—you can’t expect me to believe 3. The burden of proof is on whoever’s asking for money for a new risk-mitigating venture, to give strong evidence that the risk is real.
So you think a danger needs to likely arrive in a few decades for it to merit attention?
I think that is quite irresponsible. No law of physics states that all problems can certainly be solved very well in a few decades (the solutions for some problems might even necessarily involve political components, btw), so starting preparations earlier can be necessary.
I see “burden of proof” as a misconcept in the same way that someone “deserving” something is. A better way of thinking about this: “You seem to be making a strong claim. Mind sharing the evidence for your claim for me? …I disagree that the evidence you present justifies your claim.”
For what it’s worth, I also see “must _” as a misconcept—although “must _ to _” is not. It’s an understandable usage if the “to _*” clause is implicit, but that doesn’t seem true in this case. So to fix up SIAI’s argument, you could say that these are the statements whose probabilities are being contested:
If SarahC takes action Y before the development of Technology X and Technology X is developed, the expected value of her action will exceed its cost.
Technology X will be developed.
And depending on their probabilities, the following may or may not be true:
SarahC wants to take action Y.
Pretty much anything you say that’s not relevant to one of statements 1 or 2 (including statements that certain people haven’t been “responsible” enough in supporting their claims) is completely irrelevant to the question of whether you want to take action Y. You already have (or ought to be able to construct) probability estimates for each of 1 and 2.
Your grasp of decision theory is rather weak if you are suggesting that when Technology X is developed is irrelevant to SarahC’s decision. Similarly, you seem to suggest that the ratio of value to cost is irrelevant and that all that matters is which is bigger. Wrong again.
But your real point was not to set up a correct decision problem, but rather to suggest that her questions about whether “certain people” have been “responsible” are irrelevant. Well, I have to disagree. If action Y is giving money to “certain people”, then their level of “responsibility” is very relevant.
I did enjoy your observations regarding “burden of proof” and “must”, though probably not as much as you did.
Of course that is important. I didn’t want to include a lot of qualifiers.
I’m not trying to make a bulletproof argument so much as concisely give you an idea of why I think SarahC’s argument is malformed. My thinking is that should be enough for intellectually honest readers, as I don’t have important insights to offer beyond the concise summary. If you think I ought to write longer posts with more qualifications for readers who aren’t good at taking ideas seriously feel free to say that.
Really? So in some circumstances it is rational to take an action for which the expected cost is greater than the expected value? Or it is irrational to take an action for which the expected value exceeds the expected cost? (I’m using “rational” to mean “expected utility maximizing”, “cost” to refer to negative utility, and “value” to refer to positive utility—hopefully at this point my thought process is transparent.)
It would be a well-formed argument to say that because SIAI folks make strong claims without justifying them, they won’t use money SarahC donates well. As far as I can tell, SarahC has not explicitly made that argument. (Recall I said that she might have a correct argument in her mind but she isn’t giving us all the pieces.)
Please no insults, this isn’t you versus me is it?
No, your error was in the other direction. If you look back carefully, you will notice that the ratio is being calculated conditionally on Technology X being developed. Given that the cost is sunk regardless of whether the technology appears, it is possible that SarahC should not act even though the (conditionally) expected return exceeds the cost.
Shouldn’t be. Nor you against her. I was catty only because I imagined that you were being catty. If you were not, then I surely apologize.
I edited my post before I saw your response :-P
I’m sorry, I don’t see any edits that matter for the logic of the thread. What am I missing?
OK, my mistake.
I didn’t say what SarahC should do with the probabilities once she had them. All I said was that they were pretty much all was relevant to the question of whether she should donate. Unless I didn’t, in which case I meant to.
I’m not sure what you refer to by “actual AI.” There is a sub-field of academic computer science which calls itself “Artificial Intelligence,” but it’s not clear that this is anything more than a label, or that this field does anything more than use clever machine learning techniques to make computer programs accomplish things that once seemed to require intelligence (like playing chess, driving a car, etc.)
I’m not sure why it is a requirement that an organization concerned with the behavior of hypothetical future engineered minds would need to be in contact with these researchers.
You have to know some of their math (some of it is interesting, some not) but this does not require getting on the phone with them and asking them to explain their math, to which of course they would tell to you to RTFM instead of calling them.
Yes, the subfield of computer science is what I’m referring to.
I’m not sure that the difference between “clever machine learning techniques” and “minds” is as hard and fast as you make it. A machine that drives a car is doing one of the things a human mind does; it may, in some cases, do it through a process that’s structurally similar to the way the human mind does it. It seems to me that machines that can do these simple cognitive tasks are the best source of evidence we have today about hypothetical future thinking machines.
I gave the wrong impression here. I actually think that machine learning might be a good framework for thinking about how parts of the brain work, and I am very interested in studying machine learning. But I am skeptical that more than a small minority of projects where machine learning techniques have been applied to solve some concrete problem have shed any light on how (human) intelligence works.
In other words, I largely agree with Ben Goertzel’s assertion that there is a fundamental difference between “narrow AI” and AI research that might eventually lead to machines capable of cognition, but I’m not sure I have good evidence for this argument.
Although one should be very, very careful not to confuse the opinions of someone like Goertzel with those of the people (currently) at SIAI, I think it’s fair to say that most of them (including, in particular, Eliezer) hold a view similar to this. And this is the location—pretty much the only important one—of my disagreement with those folks. (Or, rather, I should say my differing impression from those folks—to make an important distinction brought to my attention by one of the folks in question, Anna Salamon.) Most of Eliezer’s claims about the importance of FAI research seem obviously true to me (to the point where I marvel at the fuss that is regularly made about them), but the one that I have not quite been able to swallow is the notion that AGI is only decades away, as opposed to a century or two. And the reason is essentially disagreement on the above point.
At first glance this may seem puzzling, since, given how much more attention is given to narrow AI by researchers, you might think that someone who believes AGI is “fundamentally different” from narrow AI might be more pessimistic about the prospect of AGI coming soon than someone (like me) who is inclined to suspect that the difference is essentially quantitative. The explanation, however, is that (from what I can tell) the former belief leads Eliezer and others at SIAI to assign (relatively) large amounts of probability mass to the scenario of a small set of people having some “insight” which allows them to suddenly invent AGI in a basement. In other words, they tend to view AGI as something like an unsolved math problem, like those on the Clay Millennium list, whereas it seems to me like a daunting engineering task analogous to colonizing Mars (or maybe Pluto).
This—much more than all the business about fragility of value and recursive self-improvement leading to hard takeoff, which frankly always struck me as pretty obvious, though maybe there is hindsight involved here—is the area of Eliezer’s belief map that, in my opinion, could really use more public, explicit justification.
I don’t think this is a good analogy. The problem of colonizing Mars is concrete. You can make a TODO list; you can carve the larger problem up into subproblems like rockets, fuel supply, life support, and so on. Nobody knows how to do that for AI.
OK, but it could still end up being like colonizing Mars if at some point someone realizes how to do that. Maybe komponisto thinks that someone will probably carve AGI in to subproblems before it is solved.
Well, it seems we disagree. Honestly, I see the problem of AGI as the fairly concrete one of assembling an appropriate collection of thousands-to-millions of “narrow AI” subcomponents.
Perhaps another way to put it would be that I suspect the Kolmogorov complexity of any AGI is so high that it’s unlikely that the source code could be stored in a small number of human brains (at least the way the latter currently work).
EDIT: When I say “I suspect” here, of course I mean “my impression is”. I don’t mean to imply that I don’t think this thought has occurred to the people at SIAI (though it might be nice if they could explain why they disagree).
The portion of the genome coding for brain architecture is a lot smaller than Windows 7, bit-wise.
An oddly somewhat relevant article on the information needed for specifying the brain. It is a biologist tearing a strip out of kurzweil for suggesting that we’ll be able reverse engineer the human brain in a decade by looking at the genome.
P.Z. is misreading a quote from a secondhand report. Kurzweil is not talking about reading out the genome and simulating the brain from that, but about using improvements in neuroimaging to inform input-output models of brain regions. The genome point is just an indicator of the limited number of component types involved, which helps to constrain estimates of difficulty.
Edit: Kurzweil has now replied, more or less along the lines above.
Kurzweil’s analysis is simply wrong. Here’s the gist of my refutation of it:
“So, who is right? Does the brain’s design fit into the genome? - or not?
The detailed form of proteins arises from a combination of the nucleotide sequence that specifies them, the cytoplasmic environment in which gene expression takes place, and the laws of physics.
We can safely ignore the contribution of cytoplasmic inheritance—however, the contribution of the laws of physics is harder to discount. At first sight, it may seem simply absurd to argue that the laws of physics contain design information relating to the construction of the human brain. However there is a well-established mechanism by which physical law may do just that—an idea known as the anthropic principle. This argues that the universe we observe must necessarily permit the emergence of intelligent agents. If that involves a coding the design of the brains of intelligent agents into the laws of physics then: so be it. There are plenty of apparently-arbitrary constants in physics where such information could conceivably be encoded: the fine structure constant, the cosmological constant, Planck’s constant—and so on.
At the moment, it is not even possible to bound the quantity of brain-design information so encoded. When we get machine intelligence, we will have an independent estimate of the complexity of the design required to produce an intelligent agent. Alternatively, when we know what the laws of physics are, we may be able to bound the quantity of information encoded by them. However, today neither option is available to us.”
http://alife.co.uk/essays/how_long_before_superintelligence/
Wired really messed up the flow of the talk in that case. Is it based off a singularity summit talk?
I agree with your analysis, but I also understand where PZ is coming from. You write above that the portion of the genome coding for the brain is small. PZ replies that the small part of the genome you are referring to does not by itself explain the brain; you also need to understand the decoding algorithm—itself scattered through the whole genome and perhaps also the zygotic “epigenome”. You might perhaps clarify that what you were talking about with “small portion of the genome” was the Kolmogorov complexity, so you were already including the decoding algorithm in your estimate.
The problem is, how do you get the point through to PZ and other biologists who come at the question from an evo-devo PoV? I think that someone ought to write a comment correcting PZ, but in order to do so, the commenter would have to speak the languages of three fields—neuroscience, evo-devo, and information-theory. And understand all three well enough to unpack the jargon to laymen without thereby loosing credibility with people who do know one or more of the three fields.
Why bother? PZ’s rather misguided rant isn’t doing very much damage. Just ignore him, I figure.
Maybe it is a slow news day. PZ’s rant got Slashdotted:
http://science.slashdot.org/story/10/08/17/1536233/Ray-Kurzweil-Does-Not-Understand-the-Brain
PZ has stooped pretty low with the publicity recently:
http://scienceblogs.com/pharyngula/2010/08/the_eva_mendes_sex_tape.php
Maybe he was trolling with his Kurzweil rant. He does have a history with this subject matter, though:
http://scienceblogs.com/pharyngula/2009/02/singularly_silly_singularity.php
Obviously the genome alone doesn’t build a brain. I wonder how many “bits” I should add on for the normal environment that’s also required (in terms of how much additional complexity is needed to get the first artificial mind that can learn about the world given additional sensory-like inputs). Probably not too many.
Thanks, this is useful to know. Will revise beliefs accordingly.
What do you think you know and how do you think you know it? Let’s say you have a thousand narrow AI subcomponents. (Millions = implausible due to genome size, as Carl Shulman points out.) Then what happens, besides “then a miracle occurs”?
What happens is that the machine has so many different abilities (playing chess and walking and making airline reservations and...) that its cumulative effect on its environment is comparable to a human’s or greater; in contrast to the previous version with 900 components, which was only capable of responding to the environment on the level of a chess-playing, web-searching squirrel.
This view arises from what I understand about the “modular” nature of the human brain: we think we’re a single entity that is “flexible enough” to think about lots of different things, but in reality our brains consist of a whole bunch of highly specialized “modules”, each able to do some single specific thing.
Now, to head off the “Fly Q” objection, Iet me point out that I’m not at all suggesting that an AGI has to be designed like a human brain. Instead, I’m “arguing” (expressing my perception) that the human brain’s general intelligence isn’t a miracle: intelligence really is what inevitably happens when you string zillions of neurons together in response to some optimization pressure. And the “zillions” part is crucial.
(Whoever downvoted the grandparent was being needlessly harsh. Why in the world should I self-censor here? I’m just expressing my epistemic state, and I’ve even made it clear that I don’t believe I have information that SIAI folks don’t, or am being more rational than they are.)
If a thousand species in nature with a thousand different abilities were to cooperate, would they equal the capabilities of a human? If not, what else is missing?
Tough problem. My first reaction is ‘yes’, but I think that might be because we’re assuming cooperation, which might be letting more in the door than you want.
Exactly the thought I had. Cooperation is kind of a big deal.
Yes, if there were a sufficiently powerful optimization process controlling the form of their cooperation.
I am highly confused about the parent having been voted down, to the point where I am in a state of genuine curiosity about what went through the voter’s mind as he or she saw it.
Eliezer asked whether a thousand different animals cooperating could have the power of a human. I answered:
And then someone came along, read this, and thought....what? Was it:
“No, you idiot, obviously no optimization process could be that powerful.” ?
“There you go: ‘sufficiently powerful optimization process’ is equivalent to ‘magic happens’. That’s so obvious that I’m not going to waste my time pointing it out; instead, I’m just going to lower your status with a downvote.” ?
“Clearly you didn’t understand what Eliezer was asking. You’re in over your head, and shouldn’t be discussing this topic.” ?
Something else?
Do you expect the conglomerate entity to be able to read or to be able to learn how to? Considering Eliezer can quite happily pick many many things like archer fish (ability to shoot water to take out flying insects) and chameleons (ability to control eyes independently), I’m not sure how they all add up to reading.
The optimization process is the part where the intelligence lives.
Natural selection is an optimization process, but it isn’t intelligent.
Also, the point here is AI—one is allowed to assume the use of intelligence in shaping the cooperation. That’s not the same as using intelligence as a black box in describing the nature of it.
If you were the downvoter, might I suggest giving me the benefit of the doubt that I’m up to speed on these kinds of subtleties? (I.e. if I make a comment that sounds dumb to you, think about it a little more before downvoting?)
You were at +1 when I downvoted, so I’m not alone.
Natural selection is a very bad optimization process, and so it’s quite unintelligent relative to any standards we might have as humans.
Now it’s my turn to downvote, on the grounds that you didn’t understand my comment. I agree that natural selection is unintelligent—that was my whole point! It was intended as a counterexample to your implied assertion that an appeal to an optimization process is an appeal to intelligence.
EDIT: I suppose this confirms on a small scale what had become apparent in the larger discussion here about SIAI’s public relations: people really do have more trouble noticing intellectual competence than I tend to realize.
(N.B. I just discovered that I had not, in fact, downvoted the comment that began this discussion. I must have had it confused with another.)
Like Eliezer, I generally think of intelligence and optimization as describing the same phenomenon. So when I saw this exchange:
I read your reply as meaning approximately “1000 small cognitive modules are a really powerful optimization process if and only if their cooperation is controlled by a sufficiently powerful optimization process.”
To answer the question you asked here, I thought the comment was worthy of a downvote (though apparently I did not actually follow through) because it was circular in a non-obvious way that contributed only confusion.
I am probably a much more ruthless downvoter than many other LessWrong posters; my downvotes indicate a desire to see “fewer things like this” with a very low threshold.
Thank you for explaining this, and showing that I was operating under the illusion of transparency.
My intended meaning was nothing so circular. The optimization process I was talking about was the one that would have built the machine, not something that would be “controlling” it from inside. I thought (mistakenly, it appears) that this would be clear from the fact that I said “controlling the form of their cooperation” rather than “controlling their cooperation”. My comment was really nothing different from thomblake’s or wedrifid’s. I was saying, in effect, “yes, on the assumption that the individual components can be made to cooperate, I do believe that it is possible to assemble them in so clever a manner that their cooperation would produce effective intelligence.”
The “cleverness” referred to in the previous sentence is that of the whatever created the machine (which could be actual human programmers, or, theoretically, something else like natural selection) and not the “effective intelligence” of the machine itself. (Think of a programmer, not a homunculus.) Note that I easily envision the process of implementing such “cleverness” itself not looking particularly clever—perhaps the design would be arrived at after many iterations of trial-and-error, with simpler devices of similar form. (Natural selection being the extreme case of this kind of process.) So I’m definitely not thinking magically here, and least not in any obvious way (such as would warrant a downvote, for example).
I can now see how my words weren’t as transparent as I thought, and thank you for drawing this to my attention; at the same time, I hope you’ve updated your prior that a randomly selected comment of mine results from a lack of understanding of basic concepts.
Consider me updated. Thank you for taking my brief and relatively unhelpful comments seriously, and for explaining your intended point. While I disagree that the swiftest route to AGI will involve lots of small modules, it’s a complicated topic with many areas of high uncertainty; I suspect you are at least as informed about the topic as I am, and will be assigning your opinions more credence in the future.
Hooray for polite, respectful, informative disagreements on LW!
It’s why I keep coming back even after getting mad at the place.
(That, and the fact that this is one of very few places I know where people reliably get easy questions right.)
Downvoted for retaliatory downvoting; voted everything else up toward 0.
Downvoted the parent and upvoted the grandparent. “On the grounds that you didn’t understand my comment” is a valid reason for downvoting and based off a clearly correct observation.
I do agree that komponisto would have been better served by leaving off mention of voting altogether. Just “You didn’t understand my comment. …” would have conveyed an appropriate level of assertiveness to make the point. That would have avoided sending a signal of insecurity and denied others the invitation to judge.
Voted down all comments that talk about voting, for being too much about status rather than substance.
Vote my comment towards −1 for consistency.
Status matters; it’s a basic human desideratum, like food and sex (in addition to being instrumentally useful in various ways). There seems to be a notion among some around here that concern with status is itself inherently irrational or bad in some way. But this is as wrong as saying that concern with money or good-tasting food is inherently irrational or bad. Yes, we don’t want the pursuit of status to interfere with our truth-detecting abilities; but the same goes for the pursuit of food, money, or sex, and no one thinks it’s wrong for aspiring rationalists to pursue those things. Still less is it considered bad to discuss them.
Comments like the parent are disingenuous. If we didn’t want users to think about status, we wouldn’t have adopted a karma system in the first place. A norm of forbidding the discussion of voting creates the wrong incentives: it encourages people to make aggressive status moves against others (downvoting) without explaining themselves. If a downvote is discussed, the person being targeted at least has better opportunity to gain information, rather than simply feeling attacked. They may learn whether their comment was actually stupid, or if instead the downvoter was being stupid. When I vote comments down I usually make a comment explaining why—certainly if I’m voting from 0 to −1. (Exceptions for obvious cases.)
I really don’t appreciate what you’ve done here. A little while ago I considered removing the edit from my original comment that questioned the downvote, but decided against it to preserve the context of the thread. Had I done so I wouldn’t now be suffering the stigma of a comment at −1.
Then you must be making a lot of exceptions, or you don’t downvote very much. I find that “I want to see fewer comments like this one” is true of about 1⁄3 of the comments or so, though I don’t downvote quite that much anymore since there is a cap now. Could you imagine if every 4th comment in ‘recent comments’ was taken up by my explanations of why I downvoted a comment? And then what if people didn’t like my explanations and were following the same norm—we’d quickly become a site where most comments are explaining voting behavior.
A bit of a slippery slope argument, but I think it is justified—I can make it more rigorous if need be.
Indeed I don’t downvote very much; although probably more than you’re thinking, since I on reflection I don’t typically explain my votes if they don’t affect the sign of the comment’s score.
I think you downvote too much. My perception is that, other than the rapid downvoting of trolls and inane comments, the quality of this site is the result mainly of the incentives created by upvoting, rather than downvoting.
Yes, too much explanation would also be bad; but jimrandomh apparently wants none, and I vigorously oppose that. The right to inquire about a downvote should not be trampled upon!
I have no problem with your right to inquire about a downvote; I will continue to exercise my right to downvote such requests without explanation.
I consider that a contradiction.
From the recent welcome post (emphasis added):
Perhaps we have different ideas of what ‘rights’ and ‘trampling upon’ rights entail.
You have the right to comment about reasons for downvoting—no one will stop you and armed guards will not show up and beat you for it. I think it is a good thing that you have this right.
If I think we would be better off with fewer comments like that, I’m fully within my rights to downvote the comment; similarly, no one will stop me and armed guards will not show up and beat me for it. I think it is a good thing that I have this right.
I’m not sure in what sense you think there is a contradiction between those two things, or if we are just talking past each other.
I think you should be permitted to downvote as you please, but do note that literal armed guards are not necessary for there to be real problems with the protection of rights.
My implicit premise was that 1) violent people or 2) a person actually preventing your action are severally necessary for there to be real problems with the protection of rights. Is there a problem with that version?
In such a context, when someone speaks of the “right” to do X, that means the ability to do X without being punished (in whatever way is being discussed). Here, downvoting is the analogue of armed guards beating one up.
Responding by pointing out that a yet harsher form of punishment is not being imposed is not a legitimate move, IMHO.
*reads through subthread*
You are all talking about this topic, and yet you regard me as weird??? That’s like the extrusion die asserting that the metal wire has a grey spectral component!
(if it could communicate, I mean)
It is unfortunate that I can only vote you up here once.
Ah, I could see how you would see that as a contradiction, then.
In that case, for purposes of this discussion, I withdraw my support for your right to do that.
And since I intend to downvote any comment or post for any reason I see fit at the time, it follows that no one has the right to post any comment or post of any sort, by your definition, since they can reasonably expect to be ‘punished’ for it.
For the purposes of other discussions, I do not accept your definition of ‘right’, nor do I accept your framing of a downvote as a ‘punishment’ in the relevant sense. I will continue to do my very best to ensure that only the highest-quality content is shown to new users, and if you consider that ‘punishment’, that is irrelevant to me.
I won’t bother trying any further to convince you here; but in general I will continue to ask that people behave in a less hostile manner.
Wouldn’t that analogue better apply to publicly and personally insulting the poster, targeting your verbal abuse at the very attributes that this community holds dear, deleting posts and threatening banning? Although I suppose your analogous scale could be extended in scope to include ‘imprisonment and torture without trial’.
On the topic of the immediate context I do hope that you consider thomblakes position and make an exception to your usual policy in his case. I imagine it would be extremely frustrating for you to treat others with what you consider to be respect and courtesy when you know that the recipient does not grant you the same right. It would jar with my preference for symmetry if I thought you didn’t feel free to implement a downvote friendly voting policy at least on a case by case basis. I wouldn’t consider you to be inconsistent, and definitely not hypocritical. I would consider you sane.
The proper reason to request clarification is in order to not make the mistake again—NOT as a defensive measure against some kind of imagined slight on your social status. Yes social status is a part of the reason for the karma system—but it is not something you have an inherent right to. Otherwise there would be no point to it!
Some good reasons to be downvoted: badly formed assertions, ambiguous statements, being confidently wrong, being belligerent, derailing the topic.
In this case your statement was a vague disagreement with the intuitively correct answer, with no supporting argument provided. That is just bad writing, and I would downvote it for so being. It does not imply that I think you have no real idea (something I have no grounds to take a position on), just that the specific comment did not communicate your idea effectively. You should value such feedback, as it will help you improve your writing skills,
I reject out of hand any proposed rule of propriety that stipulates people must pretend to be naive supplicants.
When people ask me for an explanation of a downvote I most certainly do not take it for granted that by so doing they are entering into my moral reality and willing to accept my interpretation of what is right and what is a ‘mistake’. If I choose to explain reasons for a downvote I also don’t expect them to henceforth conform to my will. They can choose to keep doing whatever annoying thing they were doing (there are plenty more downvotes where that one came from.)
There is more than one reason to ask for clarification for a downvote—even “I’m just kinda curious” is a valid reason. Sometimes votes just seem bizarre and not even Machiavellian reasoning helps explain the pattern. I don’t feel obliged to answer any such request but I do so if convenient. I certainly never begrudge others the opportunity to ask if they do so politely.
Not what Kompo was saying.
I never said anything about pretending anything. I said if you request clarification, and don’t actually need clarification, you’re just making noise. Ideally you will be downvoted for that.
Sure, but I still maintain that a request for clarification itself can be annoying and hence downvote worthy. I don’t think any comment is inherently protected or should be exempt from being downvoted.
I agree with you on these points. I downvote requests for clarification sometime—particularly if, say, the reason for the downvote is transparent or the flow conveys an attitude that jars with me. I certainly agree that people should be free to downvote freely whenever they please and for whatever reason they please—again, for me to presume otherwise would be a demand for naivety or dishonesty (typically both).
Feedback is valuable when it is informative, as the exchange with WrongBot turned out to be in the end.
Unfortunately, a downvote by itself will not typically be that informative. Sometimes it’s obvious why a comment was downvoted (in which case it doesn’t provide much information anyway); but in this case, I had no real idea, and it seemed plausible that it resulted from a misinterpretation of the comment. (As turned out to be the case.)
(Also, the slight to one’s social status represented by a downvote isn’t “imagined”; it’s tangible and numerical.)
The comment was a quick answer to a yes-no question posed to me by Eliezer. Would you have been more or less inclined to downvote it if I had written only “Yes”?
Providing information isn’t the point of downvoting, it is a means of expressing social disapproval. (Perhaps that is information in a sense, but it is more complicated than just that.) The fact that they are being contrary to a social norm may or may not be obvious to the commenter, if not then it is new information. Regardless, the downvote is a signal to reexamine the comment and think about why it was not approved by over 50% of the readers who felt strongly enough to vote on it.
Tangibility and significance are completely different matters. A penny might appear more solid than a dollar, but is far less worthy of consideration. You could ignore a minus-1 comment quite safely without people deciding (even momentarily) that you are a loser or some such. That you chose not to makes it look like you have an inflated view of how significant it is.
Probably less, as I would then have simply felt like requesting clarification, or perhaps even thinking of a reason on my own. A bad argument (or one that sounds bad) is worse than no argument.
You can live without sex, you can’t live without food. So the latter two are “desiderata” in rather different senses.
Status is an inherently zero-sum good, so while it is rational for any given individual to pursue it; we’d all be better off, cet par, if nobody pursued it. Everyone has a small incentive for other people not to pursue status, just as they have an incentive for them not to be violent or to smell funny; hence the existence of popular anti-status-seeking norms.
I don’t think I agree, at least in the present context. I think of status as being like money—or, in fact, the karma score on LW, since that is effectively what we’re talking about here anyway. It controls the granting of important privileges, such as what we might call “being listened to”—having folks read your words carefully, interpret them charitably, and perhaps even act on them or otherwise be influenced by them.
(To tie this to the larger context, this is why I started paying attention to SIAI: because Eliezer had won “status” in my mind.)
I agree with this.
While status may appear zero-sum amongst those who are competing for influence in a community, for the community as a whole, status is postive sum when in it accurately reflects the value of people to the community.
I don’t think I agree, at least in the present context. I think of status as being like money—or, in fact, the karma score on LW, since that is effectively what we’re talking about here anyway. It controls the granting of important privileges, such as what we might call “being listened to”—having folks read your words carefully, interpret them charitably, and perhaps even act on them or otherwise be influenced by them.
(To tie this to the larger context, this is why I started paying attention to SIAI: because Eliezer had won “status” in my mind.)
The brain has many different components with specializations, but the largest and human dominant portion, the cortex, is not really specialized at all in the way you outline.
The cortex is no more specialized than your hard drive.
Its composed of a single repeating structure and associated learning algorithm that appears to be universal. The functional specializations that appear in the adult brain arise due to topological wiring proximity to the relevant sensory and motor connections. The V1 region is not hard-wired to perform mathematically optimal gabor-like edge filters. It automatically evolves into this configuration because it is the optimal configuration for modelling the input data at that layer, and it evolves thus soley based on exposure to said input data from retinal ganglion cells.
You can think of cortical tissue as a biological ‘neuronium’. It has a semi-magical emergent capacity to self-organize into an appropriate set of feature detectors based on what its wired to. more on this
All that being said, the inter-regional wiring itself is currently less understood and is probably more genetically predetermined.
There may be other approaches that are significantly simpler (that we haven’t yet found, obviously). Assuming AGI happens, it will have been a race between the specific (type of) path you imagine, and every other alternative you didn’t think of. In other words, you think you have an upper bound on how much time/expense it will take.
I’m not a member of SIAI but my reason for thinking that AGI is not just going to be like lots of narrow bits of AI stuck together is that I can see interesting systems that haven’t been fully explored (due to difficulty of exploration). These types of systems might solve some of the open problems not addressed by narrow AI.
These are problems such as
How can a system become good at so many different things when it starts off the same. Especially puzzling is how people build complex (unconscious) machinery for dealing with problems that we are not adapted for, like Chess.
How can a system look after/upgrade itself without getting completely pwned by malware (We do get partially pwned by hostile memes, but is not complete take over of the same type as getting rooted).
Now I also doubt that these systems will develop quickly when people get around to investigating them. And they will have elements of traditional narrow AI in as well, but they will be changeable/adaptable parts of the system, not fixed sub-components. What I think needs is exploring is primarily changes in software life-cycles rather than a change in the nature of the software itself.
Learning is the capacity to build complex unconscious machinery for dealing with novel problems. Thats the whole point of AGI.
And Learning is equivalent to absorbing memes. The two are one and the same.
I don’t agree. Meme absorption is just one element of learning.
To learn how to play darts well you absorb a couple of dozen memes and then spend hours upon hours rewiring your brain to implement a complex coordination process.
To learn how to behave appropriately in a given culture you learn a huge swath of existing memes, continue to learn a stream of new ones but also dedicate huge amounts of background processing reconfiguring the weightings of existing memes relative to each other and external inputs. You also learn all sorts of implicit information about how memes work for you specifically (due to, for example, physical characteristics), much of this information will never be represented in meme form.
Fine, if you take memes to be just symbolic level transferable knowledge (which, thinking it over, I agree with), then at a more detailed level learning involves several sub-processes, one of which is the rapid transfer of memes into short term memory.
I don’t think AGI in a few decades is very farfetched at all. There’s a heckuvalot of neuroscience being done right now (the Society for Neuroscience has 40,000 members), and while it’s probably true that much of that research is concerned most directly with mere biological “implementation details” and not with “underlying algorithms” of intelligence, it is difficult for me to imagine that there will still be no significant insights into the AGI problem after 3 or 4 more decades of this amount of neuroscience research.
Of course there will be significant insights into the AGI problem over the coming decades—probably many of them. My point was that I don’t see AGI as hard because of a lack of insights; I see it as hard because it will require vast amounts of “ordinary” intellectual labor.
I’m having trouble understanding how exactly you think the AGI problem is different from any really hard math problem. Take P != NP, for instance the attempted proof that’s been making the rounds on various blogs. If you’ve skimmed any of the discussion you can see that even this attempted proof piggybacks on “vast amounts of ‘ordinary’ intellectual labor,” largely consisting of mapping out various complexity classes and their properties and relations. There’s probably been at least 30 years of complexity theory research required to make that proof attempt even possible.
I think you might be able to argue that even if we had an excellent theoretical model of an AGI, that the engineering effort required to actually implement it might be substantial and require several decades of work (e.g. Von Neumann architecture isn’t suitable for AGI implementation, so a great deal of computer engineering has to be done).
If this is your position, I think you might have a point, but I still don’t see how the effort is going to take 1 or 2 centuries. A century is a loooong time. A century ago humans barely had powered flight.
By no means do I want to downplay the difficulty of P vs NP; all the same, I think we have different meanings of “vast” in mind.
The way I think about it is: think of all the intermediate levels of technological development that exist between what we have now and outright Singularity. I would only be half-joking if I said that we ought to have flying cars before we have AGI. There are of course more important examples of technologies that seem easier than AGI, but which themselves seem decades away. Repair of spinal cord injuries; artificial vision; useful quantum computers (or an understanding of their impossibility); cures for the numerous cancers; revival of cryonics patients; weather control. (Some of these, such as vision, are arguably sub-problems of AGI: problems that would have to be solved in the course of solving AGI.)
Actually, think of math problems if you like. Surely there are conjectures in existence now—probably some of them already famous—that will take mathematicians more than a century from now to prove (assuming no Singularity or intelligence enhancement before then). Is AGI significantly easier than the hardest math problems around now? This isn’t my impression—indeed, it looks to me more analogous to problems that are considered “hopeless”, like the “problem” of classifying all groups, say.
I hate to go all existence proofy on you, but we have an existence proof of a general intelligence—accidentally sneezed out by natural selection, no less, which has severe trouble building freely rotating wheels—and no existence proof of a proof of P != NP. I don’t know much about the field, but from what I’ve heard, I wouldn’t be too surprised if proving P != NP is harder than building FAI for the unaided human mind. I wonder if Scott Aaronson would agree with me on that, even though neither of us understand the other’s field? (I just wrote him an email and asked, actually; and this time remembered not to say my opinion before asking for his.)
Scott says that he thinks P != NP is easier / likely to come first.
Here an interview with Scott Aaronson:
It’s interesting that you both seem to think that your problem is easier, I wonder if there’s a general pattern there.
What I find interesting is that the pattern nearly always goes the other way: you’re more likely to think that a celebrated problem you understand well is harder than one you don’t know much about. It says a lot about both Eliezer’s and Scott’s rationality that they think of the other guy’s hard problems as even harder than their own.
Obviously not. That would be a proof of P != NP.
As for existence proof of a general intelligence, that doesn’t prove anything about how difficult it is, for anthropic reasons. For all we know 10^20 evolutions each in 10^50 universes that would in principle allow intelligent life might on average result in 1 general intelligence actually evolving.
Of course, if you buy the self-indication assumption (which I do not) or various other related principles you’ll get an update that compels belief in quite frequent life (constrained by the Fermi paradox and a few other things).
More relevantly, approaches like Robin’s Hard Step analysis and convergent evolution (e.g. octopus/bird intelligence) can rule out substantial portions of “crazy-hard evolution of intelligence” hypothesis-space. And we know that human intelligence isn’t so unstable as to see it being regularly lost in isolated populations, as we might expect given ludicrous anthropic selection effects.
I looked at Nick’s:
http://www.anthropic-principle.com/preprints/olum/sia.pdf
I don’t get it. Anyone know what is supposed to be wrong with the SIA?
We can make better guesses than that: evolution coughed up quite a few things that would be considered pretty damn intelligent for a computer program, like ravens, octopuses, rats or dolphins.
Not independently (not even cephalopods, at least completely). And we have no way of estimating the difference in difficulty between that level of intelligence and general intelligence other than evolutionary history (which for anthropic reasons could be highly untypical), and similarity in makeup, but already know that our type of nervous system is capable of supporting general intelligence, most rat level intelligences might hit fundamental architectural problems first.
We can always estimate, even with very little knowledge—we’ll just have huge error margins. I agree it is possible that “For all we know 10^20 evolutions each in 10^50 universes that would in principle allow intelligent life might on average result in 1 general intelligence actually evolving”, I would just bet on a much higher probability than that, though I agree with the principle.
The evidence that pretty smart animals exist in distant branches of the tree of life, and in different environments is weak evidence that intelligence is “pretty accessible” in evolution’s search space. It’s stronger evidence than the mere fact that we, intelligent beings, exist.
Intelligence sure. The original point was that our existence doesn’t put a meaningful upper bound on the difficultly of general intelligence. Cephalopods are good evidence that given whatever rudimentary precursors of a nervous system our common ancestor had (I know it had differentiated cells, but I’m not sure what else. I think it didn’t really have organs like higher animals, let alone anything that really qualified as a nervous system) cephalopod level intelligence is comparatively easy, having evolved independently two times. It doesn’t say anything about how much more difficult general intelligence is compared to cephalopod intelligence, nor whether whatever precursors to a nervous system our common ancestor had were unusually conductive to intelligence compared to the average of similar complex evolved beings.
If I had to guess I would assume cephalopod level intelligence within our galaxy and a number of general intelligences somewhere outside our past light cone. But that’s because I already think of general intelligence as not fantastically difficult independently of the relevance of the existence proof.
This page on the history of invertebrates) suggests that our common ancestors had bilateral symmetry, triploblastic and with hox genes.
Hox genes suggest that they both had a modular body plan of some sort. Triploblasty implies some complexity (the least complex triploblastic organism today is a flatworm).
I’d be very surprised if most recent common ancestor didn’t have neurons similar to most neurons today, as I’ve had a hard time finding out the differences between the two. A basic introduction to nervous systems suggests they are very similar.
Well, I for one strongly hope that we resolve whether P = NP before we have AI since a large part of my estimate for the probability of AI being able to go FOOM is based on how much of the complexity hierarchy collapses. If there’s heavy collapse, AI going FOOM Is much more plausible.
Well actually, after thinking about it, I’m not sure I would either. There is something special about P vs NP, from what I understand, and I didn’t even mean to imply otherwise above; I was only disputing the idea that “vast amounts” of work had already gone into the problem, for my definition of “vast”.
Scott Aaronson’s view on this doesn’t move my opinion much (despite his large contribution to my beliefs about P vs NP), since I think he overestimates the difficulty of AGI (see your Bloggingheads diavlog with him).
Awesome! Be sure to let us know what he thinks. Sounds unbelievable to me though, but what do I know.
Why is AGI a math problem? What is abstract about it?
We don’t need math proofs to know if AGI is possible. It is, the brain is living proof.
We don’t need math proofs to know how to build AGI—we can reverse engineer the brain.
There may be a few clues in there—but engineers are likely to get to the goal looong before the emulators arrive—and engineers are math-friendly.
A ‘few clues’ sounds like a gross underestimation. It is the only working example, so it certainly contains all the clues, not just a few. The question of course is how much of a shortcut is possible. The answer to date seems to be: none to slim.
I agree engineers reverse engineering will succeed way ahead of full emulation, that wasn’t my point.
If information is not extracted and used, it doesn’t qualify as being a “clue”.
The search oracles and stockmarketbot makers have paid precious little attention to the brain. They are based on engineering principles instead.
Most engineers spend very little time on reverse-engineering nature. There is a little “bioinspiration”—but inspiration is a bit different from wholescale copying.
This is a good part of the guts of it. That bit of it is a math problem:
http://timtyler.org/sequence_prediction/
I think the following quote is illustrative of the problems facing the field:
-Marvin Minsky, quoted in “AI” by Daniel Crevier.
Some notes and interpretation of this comment:
Most vision researchers, if asked who is the most important contributor to their field, would probably answer “David Marr”. He set the direction for subsequent research in the field; students in introductory vision classes read his papers first.
Edge detection is a tiny part of vision, and vision is a tiny part of intelligence, but at least in Minsky’s view, no progress (or reverse progress) was achieved in twenty years of research by the leading lights of the field.
There is no standard method for evaluating edge detector algorithms, so it is essentially impossible to measure progress in any rigorous way.
I think this kind of observation justifies AI-timeframes on the order of centuries.
Edge detection is rather trivial. Visual recognition however is not, and there certainly are benchmarks and comparable results in that field. Have you browsed the recent pubs of Poggio et al at MIT vision lab? There is lots of recent progress, with results matching human levels for quick recognition tasks.
Also, vision is not a tiny part of intelligence. Its the single largest functional component of the cortex, by far. The cortex uses the same essential low-level optimization algorithm everywhere, so understanding vision at the detailed level is a good step towards understanding the whole thing.
And finally and most relevant for AGI, the higher visual regions also give us the capacity for visualization and are critical for higher creative intelligence. Literally all scientific discovery and progress depends on this system.
“visualization is the key to enlightenment” and all that
the visual system
It’s only trivial if you define an “edge” in a trivial way, e.g. as a set of points where the intensity gradient is greater than a certain threshold. This kind of definition has little use: given a picture of a tree trunk, this definition will indicate many edges corresponding to the ridges and corrugations of the bark, and will not highlight the meaningful edge between the trunk and the background.
I don’t believe that there is much real progress recently in vision. I think the state of the art is well illustrated by the “racist” HP web camera that detects white faces but not black faces.
I actually agree with you about this, but I think most people on LW would disagree.
Whether you are talking about canny edge filters, gabor like edge detection more similar to what V1 self-organizes into, they are all still relatively simple—trivial compared to AGI. Trivial as in something you code in a few hours for your screen filter system in a modern game render engine.
The particular problem you point out with the tree trunk is a scale problem and is easily handled in any good vision system.
An edge detection filter is just a building block, its not the complete system.
In HVS, initial edge preprocessing is done in the retina itself which essentially does on-center, off-surround gaussian filters (similar to low-pass filters in photoshop). The output of the retina is thus essentially a multi-resolution image set, similar to a wavelet decomposition. The image output at this stage becomes a series of edge differences (local gradients), but at numerous spatial scales.
The high frequency edges such as the ridges and corrugations of the bark are cleanly separated from the more important low frequency edges separating the tree trunk from the background. V1 then detects edge orientations at these various scales, and higher layers start recognizing increasingly complex statistical patterns of edges across larger fields of view.
Whether there is much real progress recently in computer vision is relative to one’s expectations, but the current state of the art in research systems at least is far beyond your simplistic assessment. I have a layman’s overview of HVS here. If you really want to know about the current state of the art in research, read some recent papers from a place like Poggio’s lab at MIT.
In the product space, the HP web camera example is also very far from the state of the art, I’m surprised that you posted that.
There is free eye tracking software you can get (running on your PC) that can use your web cam to track where your eyes are currently focused in real time. That’s still not even the state of the art in the product space—that would probably be the systems used in the more expensive robots, and of course that lags the research state of the art.
...but you don’t really know—right?
You can’t say with much confidence that there’s no AIXI-shaped magic bullet.
That’s right; I’m not an expert in AI. Hence I am describing my impressions, not my fully Aumannized Bayesian beliefs.
AIXI-shaped magic bullet?
AIXI’s contribution is more philosophical than practical. I find a depressing over-emphasis of bayesian probability theory here as the ‘math’ of choice vs computational complexity theory, which is the proper domain.
The most likely outcome of a math breakthrough will be some rough lower and or upper bounds on the shape of the intelligence over space/time complexity function. And right now the most likely bet seems to be that the brain is pretty well optimized at the circuit level, and that the best we can do is reverse engineer it.
EY and the math folk here reach a very different conclusion, but I have yet to find his well considered justification. I suspect that the major reason the mainstream AI community doesn’t subscribe to SIAI’s math magic bullet theory is that they hold the same position outline above: ie that when we get the math theorems, all they will show is what we already suspect: human level intelligence requires X memory bits and Y bit ops/second, where X and Y are roughly close to brain levels.
This, if true, kills the entirety of the software recursive self-improvement theory. The best that software can do is approach the theoretical optimum complexity class for the problem, and then after that point all one can do is fix it into hardware for a further large constant gain.
I explore this a little more here
That seems like crazy talk to me. The brain is not optimal—not its hardware or software—and not by a looooong way! Computers have already steam-rollered its memory and arithmetic -units—and that happened before we even had nanotechonolgy computing components. The rest of the brain seems likely to follow.
Edit: removed a faulty argument at the end pointed out by wedrifid.
I am talking about optimality for AGI in particular with respect to circuit complexity, with the typical assumptions that a synapse is vaguely equivalent to a transistor, maybe ten transistors at most. If you compare on that level, the brain looks extremely efficient given how slow the neurons are. Does this make sense?
The brain’s circuits have around 10^15 transistor equivalents, and a speed of 10^3 cycles per second. 10^18 transistor cycles / second
A typical modern CPU has 10^9 transistors, with a speed of 10^9 cycles per second. 10^18 transistor cycles / second
Our CPU’s strength is not their circuit architecture or software—its the raw speed of CMOS, its a million X substrate advantage. The learning algorithm, the way in which the cortex rewires in response to input data, appears to be a pretty effective universal learning algorithm.
The brain’s architecture is a joke. It is as though a telecoms engineer decided to connect a whole city’s worth of people together by running cables directly between any two people who wanted to have a chat. It hasn’t even gone fully digital yet—so things can’t easily be copied or backed up. The brain is just awful—no wonder human cognition is such a mess.
Nothing you wrote lead me to this conclusion.
Then some questions: How long would moore’s law have to continue into the future with no success in AGI for that to show that the brain’s is well optimized for AGI at the circuit level?
I’ve taken some attempts to show rough bounds on the brain’s efficiency, are you aware of some other approach or estimate?
Most seem to think the problem is mostly down to software—and that supercomputer hardware is enough today—in which case more hardware would not necessarily help very much. The success or failure of adding more hardware might give an indication of how hard it is to find the target of intelligence in the search space. It would not throw much light on the issue of how optimally “designed” the brain is. So: your question is a curious one.
For every computational system and algorithm, there is a minimum level of space-time complexity in which this system can be encoded. As of yet we don’t know how close the brain is to the minimum space-time complexity design for an intelligence of similar capability.
Lets make the question more specific: whats the minimum bit representation of a human-equivalent mind? If you think the brain is far off that, how do you justify that?
Of course more hardware helps: it allows you to search through the phase space faster. Keep in mind the enormity of the training time.
I happen to believe the problem is ‘mostly down to software’, but I don’t see that as a majority view—the Moravec/Kurzweil view that we need brain-level hardware (within an order of magnitude or so) seems to be majoritive at this point.
We need brain-level hardware (within an order of magnitude or so) if machines are going to be cost-competitive with humans. If you just want a supercomputer mind, then no problem.
I don’t think Moravec or Kurzweil ever claimed it was mostly down to hardware. Moravec’s charts are of hardware capability—but that was mainly because you can easily measure that.
I don’t see why that is. If you were talking about ems, then the threshhold should be 1:1 realtime. Otherwise, for most problems that we know how to program a computer to do, the computer is much faster than humans even at existing speeds. Why do you expect that a computer that’s say, 3x slower than a human (well within an order of magnitude) would be cost-competitive with humans while one that’s 10^4 times slower wouldn’t?
Evidently there are domains where computers beat humans today—but if you look at what has to happen for machines to take the jobs of most human workers, they will need bigger and cheaper brains to do that. “Within an order of magnitude or so” seems like a reasonable ballpark figure to me. If you are looking for more details about why I think that, they are not available at this time.
I suspect that the controlling reason why you think that is that you assume it takes human-like hardware to accomplish human-like tasks, and greatly underestimate the advantages of a mind being designed rather than evolved.
Way off. Let’s see… I would bet at even odds that it is 4 or more orders of magnitude off optimal.
We have approximately one hundred billion neurons each and roughly the same number of glial cells (more of the latter if we are smart!). Each of those includes a full copy of our DNA, which is itself not exactly optimally compressed.
you didn’t answer my question: what is your guess at minimum bit representation of a human equi mind?
you didn’t use the typical methodology of measuring the brain’s storage, nor did you provide another.
I wasn’t talking about molecular level optimization. I started with the typical assumption that synapses represent a few bits, the human brain has around 100TB to 1PB of data/circuitry, etc etc—see the singularity is near.
So you say the human brain algorithmic representation is off by 4 orders of magnitude or more—you are saying that you think a human equivalent mind can be represented in 10 to 100GB of data/circuitry?
If so, why did evolution not find that by now? It has had plenty of time to compress at the circuit level. In fact, we actually know that the brain does perform provably optimal compression on its input data in a couple of domains—see V1 and its evolution into gabor-like edge feature detection.
Evolution has had plenty of time to find a well-optimized cellular machinery based on DNA, plenty of time to find a well-optimized electro-chemical computing machinery based on top of that, and plenty of time to find well-optimized circuits within that space.
Even insects are extremely well-optimized at the circuit level—given their neuron/synapse counts, we have no evidence whatsoever to believe that vastly simpler circuits exist that can perform the same functionality.
When we have used evolutionary exploration algorithms to design circuits natively, given enough time we see similar complex, messy, but near optimal designs, and this is a general trend.
Are you saying that you are counting every copy of the DNA as information that contributes to the total amount? If so, I say that’s invalid. What if each cell were remotely controlled from a central server containing the DNA information? I can’t see that we’d count the DNA for each cell then—yet it is no different really.
I agree that the number of cells is relevant, because there will be a lot of information in the structure of an adult brain that has come from the environment, rather than just from the DNA, and more cells would seem to imply more machinery in which to put it.
I thought we were talking about the efficiency of the human brain. Wasn’t that the whole point? If every cell is remotely controlled from a central server then well, that’d be whole different algorithm. In fact, we could probably scrap the brain and just run the central server.
Genes actually do matter in the functioning of neurons. Chemical additions (eg. ethanol) and changes in the environment (eg. hypoxia) can influence gene expression in cells in the brain, impacting on their function.
I suggest the brain is a ridiculously inefficient contraption thrown together by the building blocks that were practical for production from DNA representations and suitable for the kind of environments animals tended to be exposed to. We should be shocked to find that it also manages to be anywhere near optimal for general intelligence. Among other things it would suggest that evolution packed the wrong lunch.
Okay, I may have misunderstood you. It looks like there is some common ground between us on the issue of inefficiency. I think the brain would probably be inefficient as well as it has to be thrown together by the very specific kind of process of evolution—which is optimized for building things without needing look-ahead intelligence rather than achieving the most efficient results.
A Sperm Whale and a bowl of Petunias.
My first impulse was to answer that Moore’s law could go forever and never produce success in AGI, since ‘AGI’ isn’t just what you get when you put enough computronium together for it to reach critical mass. But even given no improvements in understanding we could very well arrive at AGI just through ridiculous amounts of brute force. In fact, given enough space and time, randomised initial positions and possibly a steady introduction of negentropy we could produce an AGI in Conways Life.
You could find some rough bounds by seeing how many parts of a human brain you can cut out without changing IQ.Trivial little things like, you know, the pre-frontal cortex.
You are just talking around my questions, so let me make it more concrete. An important task of any AGI is higher level sensor data interpretation—ie seeing. We have an example system in the human brain—the human visual system, which is currently leaps and bounds beyond the state of the art in machine vision. (although the latter is making progress towards the former through reverse engineering)
So machine vision is a subtask of AGI. What is the minimal computational complexity of human-level vision? This is a concrete computer science problem. It has a concrete answer—not “sperm whale and petunia” nonsense.
Until someone makes a system better than HVS, or proves some complexity bounds, we don’t know how optimal HVS is for this problem, but we also have no reason to believe that it is orders of magnitude off from the theoretic optimum.
The article linked to in the parent is entitled:
“Created in the Likeness of the Human Mind: Why Strong AI will necessarily be like us”
Good quality general-purpose data-compression would “break the back” of the task of buliding synthetic intelligent agents—and that’s a “simple” math problem—as I explain on: http://timtyler.org/sequence_prediction/
At least it can be stated very concisely. Solutions so far haven’t been very simple—but the brain’s architecture offers considerable hope for a relatively simple solution.
Note that allowing for a possibility of sudden breakthrough is also an antiprediction, not a claim for a particular way things are. You can’t know that no such thing is possible, without having understanding of the solution already at hand, hence you must accept the risk. It’s also possible that it’ll take a long time.
I’m reading through and catching up on this thread, and rather strongly agreed with your statement:
However, pondering it again, I realize there is an epistemological spectrum ranging from math on the one side to engineering on the other. Key insights into new algorithms can undoubtedly speed up progress, and such new insights often can be expressed as pure math, but at the end of the day it is a grand engineering (or reverse engineering) challenge.
However, I’m somewhat taken aback when you say, “the notion that AGI is only decades away, as opposed to a century or two.”
A century or two?
One obvious piece of evidence is that many forms of narrow learning are mathematically incapable of doing much. There are for example a whole host of theorems about what different classes of neural networks can actually recognize, and the results aren’t very impressive. Similarly, support vector machine’s have a lot of trouble learning anything that isn’t a very simple statistical model, and even then humans need to decide which stats are relevant. Other linear classifiers run into similar problems.
I work in this field, and was under approximately the opposite impression; that voice and visual recognition are rapidly approaching human levels. If I’m wrong and there are sharp limits, I’d like to know. Thanks!
Machine intelligence has surpassed “human level” in a number of narrow domains. Already, humans can’t manipulate enough data to do anything remotely like a search engine or a stockbot can do.
The claim seems to be that in narrow domains there are often domain-specific “tricks”—that wind up not having much to do with general intelligence—e.g. see chess and go. This seems true—but narrow projects often broaden out. Search engines and stockbots really need to read and understand the web. The pressure to develop general intelligence in those domains seems pretty strong.
Those who make a big deal about the distinction between their projects and “mere” expert systems are probably mostly trying to market their projects before they are really experts at anything.
One of my videos discusses the issue of whether the path to superintelligent machines will be “broad” or “narrow”:
http://alife.co.uk/essays/on_general_machine_intelligence_strategies/
Thanks, it always is good to actually have input from people who work in a given field. So please correct me if I’m wrong but I’m under the impression that
1) neutral networks cannot in general detect connected components unless the network has some form of recursion. 2) No one knows how to make a neural network with recursion learn in any effective, marginally predictable fashion.
This is the sort of thing I was thinking of. Am I wrong about 1 or 2?
Not sure what you mean about by 1), but certainly, recurrent neural nets are more powerful. 2) is no longer true; see for example the GeneRec algorithm. It does something much like backpropagation, but with no derivatives explicitly calculated, there’s no concern with recurrent loops.
On the whole, neural net research has slowed dramatically based on the common view you’ve expressed; but progress continues apace, and they are not far behind cutting edge vision and speech processing algorithms, while working much more like the brain does.
Thanks. GeneRec sounds very interesting. Will take a look. Regarding 1, I was thinking of something like the theorems in chapter 9 in Perceptrons which shows that there are strong limits on what topological features of input a non-recursive neural net can recognize.
Prediction is hard, especially about the future.
One thing that intrigues me is snags. Did anyone predict how hard to would be to improve batteries, especially batteries big enough for cars?
I agree completely. The reason why I framed my top level post in the way that I did was so that it would be relevant to readers of a variety of levels of confidence in SIAI’s claims.
As I indicate here, I personally wouldn’t be interested in funding SIAI as presently constituted even if there was no PR problem.
I think there are ways to make these predictions. On the most layman level I would point out that IBM build a robot that beats people at Jeopardy. Yes, I am aware that this is a complete machine-learning hack (this is what I could gather from the NYT coverage) and is not true cognition, but it surprised even me (I do know something about ML). I think this is useful to defeat the intuition of “machines cannot do that”. If you are truly interested I think you can (I know you’re capable) read Norvig’s AI book, and than follow up on the parts of it that most resemble human cognition; I think serious progress is made in those areas. BTW, Norvig does take FAI issues seriously, including a reference to EY paper in the book.
I think they should, I have no idea if this is being done; but if I would do it I would not do it publicly, as it may have very counterproductive consequences. So until you or I become SIAI fellows we will not know, and I cannot hold such lack of knowledge against them.
First, I’m not really claiming “machines cannot do that.” I can see advances in machine learning and I can imagine the next round of advances being pretty exciting. But I’m thinking in terms of maybe someday a machine being able to distinguish foreground from background, or understand a sentence in English, not being a superintelligence that controls Earth’s destiny. The scales are completely different. One scale is reasonable; one strains credibility, I’m afraid.
Thanks for the book recommendation; I’ll be sure to check it out.
I think controlling Earth’s destiny is only modestly harder than understanding a sentence in English—in the same sense that I think Einstein was only modestly smarter than George W. Bush. EY makes a similar point.
You sound to me like someone saying, sixty years ago: “Maybe some day a computer will be able to play a legal game of chess—but simultaneously defeating multiple grandmasters, that strains credibility, I’m afraid.” But it only took a few decades to get from point A to point B. I doubt that going from “understanding English” to “controlling the Earth” will take that long.
Well said. I shall have to try to remember that tagline.
There’s a problem with it, though. Some decades ago you’d have just as eagerly subscribed to this statement: “Controlling Earth’s destiny is only modestly harder than playing a good game of chess”, which we now know to be almost certainly false.
I agree with Rain. Understanding implies a much deeper model than playing. To make the comparison to chess, you would have to change it to something like, “Controlling Earth’s destiny is only modestly harder than making something that can learn chess, or any other board game, without that game’s mechanics (or any mapping from the computer’s output to game moves) being hard-coded, and then play it at an expert level.”
Not obviously false, I think.
It’s the word “understanding” in the quote which makes it presume general intelligence and/or consciousness without directly stating it. The word “playing” does not have such a connotation, at least to me. I don’t know if I would think differently back when chess required intelligence.
(Again:) Hey, remember this tagline: “I think controlling Earth’s destiny is only modestly harder than understanding a sentence in English.”
Hey, remember this tagline: “I think controlling Earth’s destiny is only modestly harder than understanding a sentence in English.”
Yes. It’s hardly urgent, since AI researchers are nowhere near a runaway intelligence. But on the other hand, control of AI is going to be crucial+difficult eventually, and it would be good for researchers to be aware of it, if they aren’t.
Sadly, there’s no guarantee of that.
Right, it’s just (in my and most other AI researchers’[*] opinion) overwhelmingly likely that we are in fact nowhere near (the capability of) it. Although it’s interesting to me that I don’t feel there’s that much difference in probability of “(good enough to) run away improving itself quickly past human level AI” in the next year, and in the next 10 years—both extremely close to 0 is the most specific I can be at this point. That suggests I haven’t really quantified my beliefs exactly yet.
[*] I actually only work on natural language processing using really dumb machine learning, i.e. not general AI.