kalla724

Karma: 1,257

kalla724 17 May 2012 21:51 UTC
2 points
in reply to: JoshuaZ’s comment on: Thoughts on the Singularity Institute (SI)
Yes. If we have an AGI, and someone sets forth to teach it how to be able to lie, I will get worried.

I am not worried about an AGI developing such an ability spontaneously.

kalla724 17 May 2012 21:50 UTC
6 points
in reply to: JoshuaZ’s comment on: Thoughts on the Singularity Institute (SI)
In the infinite number of possible paths, the percent of paths we are adding up to here is still very close to zero.

Perhaps I can attempt another rephrasing of the problem: what is the mechanism that would make an AI automatically seek these paths out, or make them any more likely than infinite number of other paths?

I.e. if we develop an AI which is not specifically designed for the purpose of destroying life on Earth, how would that AI get to a desire to destroy life on Earth, and by which mechanism would it gain the ability to accomplish its goal?

This entire problem seems to assume that an AI will want to “get free” or that its primary mission will somehow inevitably lead to a desire to get rid of us (as opposed to a desire to, say, send a signal consisting of 0101101 repeated an infinite number of times in the direction of Zeta Draconis, or any other possible random desire). And that this AI will be able to acquire the abilities and tools required to execute such a desire. Every time I look at such scenarios, there are abilities that are just assumed to exist or appear on their own (such as the theory of mind), which to the best of my understanding are not a necessary or even likely products of computation.

In the final rephrasing of the problem: if we can make an AGI, we can probably design an AGI for the purpose of developing an AGI that has a theory of mind. This AGI would then be capable of deducing things like deception or the need for deception. But the point is—unless we intentionally do this, it isn’t going to happen. Self-optimizing intelligence doesn’t self-optimize in the direction of having theory of mind, understanding deception, or anything similar. It could, randomly, but it also could do any other random thing from the infinite set of possible random things.

kalla724 17 May 2012 21:42 UTC
2 points
in reply to: dlthomas’s comment on: Thoughts on the Singularity Institute (SI)
You are correct. I did not phrase my original posts carefully.

I hope that my further comments have made my position more clear?

kalla724 17 May 2012 21:40 UTC
2 points
in reply to: JoshuaZ’s comment on: Thoughts on the Singularity Institute (SI)
We are trapped in an endless chain here. The computer would still somehow have to deduce that Wikipedia entry that describes One Ring is real, while the One Ring itself is not.

kalla724 17 May 2012 21:38 UTC
2 points
in reply to: Eliezer Yudkowsky’s comment on: Thoughts on the Singularity Institute (SI)
My apologies, but this is something completely different.

The scenario takes human beings—which have a desire to escape the box, possess theory of mind that allows them to conceive of notions such as “what are aliens thinking” or “deception”, etc. Then it puts them in the role of the AI.

What I’m looking for is a plausible mechanism by which an AI might spontaneously develop such abilities. How (and why) would an AI develop a desire to escape from the box? How (and why) would an AI develop a theory of mind? Absent a theory of mind, how would it ever be able to manipulate humans?

kalla724 17 May 2012 20:20 UTC
5 points
in reply to: JoshuaZ’s comment on: Thoughts on the Singularity Institute (SI)
Yet again: ability to discern which parts of fiction accurately reflect human psychology.

An AI searches the internet. It finds a fictional account about early warning systems causing nuclear war. It finds discussions about this topic. It finds a fictional account about Frodo taking the Ring to Mount Doom. It finds discussions about this topic. Why does this AI dedicate its next 10^15 cycles to determination of how to mess with the early warning systems, and not to determination of how to create One Ring to Rule them All?

(Plus other problems mentioned in the other comments.)

kalla724 17 May 2012 20:17 UTC
3 points
in reply to: Mass_Driver’s comment on: Thoughts on the Singularity Institute (SI)
Doesn’t work.

This requires the AI to already have the ability to comprehend what manipulation is, to develop manipulation strategy of any kind (even one that will succeed 0.01% of the time), ability to hide its true intent, ability to understand that not hiding its true intent would be bad, and the ability to discern which issues are low-salience and which high-salience for humans from the get-go. And many other things, actually, but this is already quite a list.

None of these abilities automatically “fall out” from an intelligent system either.

kalla724 17 May 2012 20:14 UTC
5 points
in reply to: JoshuaZ’s comment on: Thoughts on the Singularity Institute (SI)

This seems like an accurate and a highly relevant point. Searching a solution space faster doesn’t mean one can find a better solution if it isn’t there.

Or if your search algorithm never accesses relevant search space. Quantitative advantage in one system does not translate into quantitative advantage in a qualitatively different system.

kalla724 17 May 2012 20:12 UTC
4 points
in reply to: JoshuaZ’s comment on: Thoughts on the Singularity Institute (SI)
That is my point: it doesn’t get to find out about general human behavior, not even from the Internet. It lacks the systems to contextualize human interactions, which have nothing to do with general intelligence.

Take a hugely mathematically capable autistic kid. Give him access to the internet. Watch him develop ability to recognize human interactions, understand human priorities, etc. to a sufficient degree that it recognizes that hacking an early warning system is the way to go?

kalla724 17 May 2012 20:09 UTC
3 points
in reply to: A1987dM’s comment on: Thoughts on the Singularity Institute (SI)
Only if it has the skills required to analyze and contextualize human interactions. Otherwise, the Internet is a whole lot of jibberish.

Again, these skills do not automatically fall out of any intelligent system.

kalla724 17 May 2012 19:00 UTC
5 points
in reply to: JoshuaZ’s comment on: Thoughts on the Singularity Institute (SI)
I’m not so much interested in the exact mechanism of how humans would be convinced to go to war, as in an even approximate mechanism by which an AI would become good at convincing humans to do anything.

Ability to communicate a desire and convince people to take a particular course of action is not something that automatically “falls out” from an intelligent system. You need a theory of mind, an understanding of what to say, when to say it, and how to present information. There are hundreds of kids on autistic spectrum who could trounce both of us in math, but are completely unable to communicate an idea.

For an AI to develop these skills, it would somehow have to have access to information on how to communicate with humans; it would have to develop the concept of deception; a theory of mind; and establish methods of communication that would allow it to trick people into launching nukes. Furthermore, it would have to do all of this without trial communications and experimentation which would give away its goal.

Maybe I’m missing something, but I don’t see a straightforward way something like that could happen. And I would like to see even an outline of a mechanism for such an event.

kalla724 17 May 2012 18:00 UTC
5 points
in reply to: jacob_cannell’s comment on: Thoughts on the Singularity Institute (SI)
I’m vaguely familiar with the models you mention. Correct me if I’m wrong, but don’t they have a final stopping point, which we are actually projected to reach in ten to twenty years? At a certain point, further miniaturization becomes unfeasible, and the growth of computational power slows to a crawl. This has been put forward as one of the main reasons for research into optronics, spintronics, etc.

We do NOT have sufficient basic information to develop processors based on simulation alone in those other areas. Much more practical work is necessary.

As for point 2, can you provide a likely mechanism by which a FOOMing AI could detonate a large number of high-yield thermonuclear weapons? Just saying “human servitors would do it” is not enough. How would the AI convince the human servitors to do this? How would it get access to data on how to manipulate humans, and how would it be able to develop human manipulation techniques without feedback trials (which would give away its intention)?

kalla724 17 May 2012 17:55 UTC
7 points
in reply to: Bugmaster’s comment on: Thoughts on the Singularity Institute (SI)
By all means, continue. It’s an interesting topic to think about.

The problem with “atoms up” simulation is the amount of computational power it requires. Think about the difference in complexity when calculating a three-body problem as compared to a two-body problem?

Than take into account the current protein folding algorithms. People have been trying to calculate folding of single protein molecules (and fairly short at that) by taking into account the main physical forces at play. In order to do this in a reasonable amount of time, great shortcuts have to be taken—instead of integrating forces, changes are treated as stepwise, forces beneath certain thresholds are ignored, etc. This means that a result will always have only a certain probability of being right.

A self-replicating nanomachine requires minimal motors, manipulators and assemblers; while still tiny, it would be a molecular complex measured in megadaltons. To precisely simulate creation of such a machine, an AI that is trillion times faster than all the computers in the world combined would still require decades, if not centuries of processing time. And that is, again, assuming that we know all the forces involved perfectly, which we don’t (how will microfluidic effects affect a particular nanomachine that enters human bloodstream, for example?).

kalla724 17 May 2012 5:26 UTC
0 points
in reply to: Bugmaster’s comment on: Thoughts on the Singularity Institute (SI)
See my answer to dlthomas.

kalla724 17 May 2012 5:25 UTC
8 points
in reply to: dlthomas’s comment on: Thoughts on the Singularity Institute (SI)
Scaling it up is absolutely dependent on currently nonexistent information. This is not my area, but a lot of my work revolves around control of kinesin and dynein (molecular motors that carry cargoes via microtubule tracks), and the problems are often similar in nature.

Essentially, we can make small pieces. Putting them together is an entirely different thing. But let’s make this more general.

The process of discovery has, so far throughout history, followed a very irregular path. 1- there is a general idea 2- some progress is made 3- progress runs into an unpredicted and previously unknown obstacle, which is uncovered by experimentation. 4- work is done to overcome this obstacle. 5- goto 2, for many cycles, until a goal is achieved—which may or may not be close to the original idea.

I am not the one who is making positive claims here. All I’m saying is that what has happened before is likely to happen again. A team of human researchers or an AGI can use currently available information to build something (anything, nanoscale or macroscale) to the place to which it has already been built. Pushing it beyond that point almost invariably runs into previously unforeseen problems. Being unforeseen, these problems were not part of models or simulations; they have to be accounted for independently.

A positive claim is that an AI will have a magical-like power to somehow avoid this—that it will be able to simulate even those steps that haven’t been attempted yet so perfectly, that all possible problems will be overcome at the simulation step. I find that to be unlikely.

kalla724 17 May 2012 2:56 UTC
3 points
in reply to: dlthomas’s comment on: Thoughts on the Singularity Institute (SI)
With absolute certainty, I don’t. If absolute certainty is what you are talking about, then this discussion has nothing to do with science.

If you aren’t talking about absolutes, then you can make your own estimation of likelihood that somehow an AI can derive correct conclusions from incomplete data (and then correct second order conclusions from those first conclusions, and third order, and so on). And our current data is woefully incomplete, many of our basic measurements imprecise.

In other words, your criticism here seems to boil down to saying “I believe that an AI can take an incomplete dataset and, by using some AI-magic we cannot conceive of, infer how to END THE WORLD.”

Color me unimpressed.

kalla724 17 May 2012 2:24 UTC
2 points
in reply to: Bugmaster’s comment on: Thoughts on the Singularity Institute (SI)
Yes, but it can’t get to nanotechnology without a whole lot of experimentation. It can’t deduce how to create nanorobots, it would have to figure it out by testing and experimentation. Both steps limited in speed, far more than sheer computation.
What links here?
- dlthomas's comment on Thoughts on the Singularity Institute (SI) by HoldenKarnofsky (17 May 2012 21:28 UTC; 0 points)

kalla724 17 May 2012 1:49 UTC
7 points
in reply to: dlthomas’s comment on: Thoughts on the Singularity Institute (SI)
I’m not talking about limited sensory data here (although that would fall under point 2). The issue is much broader:
- We humans have limited data on how the universe work
- Only a limited subset of that limited data is available to any intelligence, real or artificial
Say that you make a FOOM-ing AI that has decided to make all humans dopaminergic systems work in a particular, “better” way. This AI would have to figure out how to do so from the available data on the dopaminergic system. It could analyze that data millions of times more effectively than any human. It could integrate many seemingly irrelevant details.

But in the end, it simply would not have enough information to design a system that would allow it to reach its objective. It could probably suggest some awesome and to-the-point experiments, but these experiments would then require time to do (as they are limited by the growth and development time of humans, and by the experimental methodologies involved).

This process, in my mind, limits the FOOM-ing speed to far below what seems to be implied by the SI.

This also limits bootstrapping speed. Say an AI develops a much better substrate for itself, and has access to the technology to create such a substrate. At best, this substrate will be a bit better and faster than anything humanity currently has. The AI does not have access to the precise data about basic laws of universe it needs to develop even better substrates, for the simple reason that nobody has done the experiments and precise enough measurements. The AI can design such experiments, but they will take real time (not computational time) to perform.

Even if we imagine an AI that can calculate anything from the first principles, it is limited by the precision of our knowledge of those first principles. Once it hits upon those limitations, it would have to experimentally produce new rounds of data.

kalla724 17 May 2012 1:11 UTC
9 points
in reply to: Eliezer Yudkowsky’s comment on: Thoughts on the Singularity Institute (SI)
Hm. I must be missing something. No, I haven’t read all the sequences in detail, so if these are silly, basic, questions—please just point me to the specific articles that answer them.

You have an Oracle AI that is, say, a trillionfold better at taking existing data and producing inferences.

1) This Oracle AI produces inferences. It still needs to test those inferences (i.e. perform experiments) and get data that allow the next inferential cycle to commence. Without experimental feedback, the inferential chain will quickly either expand into an infinity of possibilities (i.e. beyond anything that any physically possible intelligence can consider), or it will deviate from reality. The general intelligence is only as good as the data its inferences are based upon.

Experiments take time, data analysis takes time. No matter how efficient the inferential step may become, this puts an absolute limit to the speed of growth in capability to actually change things.

2) The Oracle AI that “goes FOOM” confined to a server cloud would somehow have to create servitors capable of acting out its desires in the material world. Otherwise, you have a very angry and very impotent AI. If you increase a person’s intelligence trillionfold, and then enclose them into a sealed concrete cell, they will never get out; their intelligence can calculate all possible escape solutions, but none will actually work.

Do you have a plausible scenario how a “FOOM”-ing AI could—no matter how intelligent—minimize oxygen content of our planet’s atmosphere, or any such scenario? After all, it’s not like we have any fully-automated nanobot production factories that could be hijacked.

kalla724 14 May 2012 18:38 UTC
14 points
in reply to: lsparrish’s comment on: Neil deGrasse Tyson on Cryonics
1. Yes, if you can avoid replacing the solvent. But how do you avoid that, and still avoid creation of ice crystals? Actually, now that I think of it, there is a possible solution: expressing icefish proteins within neuronal cells. Of course, who knows shat they would do to neuronal physiology, and you can’t really express them after death...
2. I’m not sure that less toxic cryoprotectants are really feasible. But yes, that would be a good step forward.
3. I actually think it’s better to keep them together. Trying theoretical approaches as quickly as possible and having an appliable goal ahead at all times are both good for the speed of progress. There is a reason science moves so much faster during times of conflict, for example.