ZoltanBerrigomo

Karma: 98

ZoltanBerrigomo 15 Jan 2016 1:11 UTC
0 points
in reply to: HungryHobo’s comment on: What can go wrong with the following protocol for AI containment?

If you limit yourself to a subset of features such that you are no longer writing in a format which is turing complete then you may be able to have a program capable of automatically proving that code reliably.

Right, that is what i meant.

ZoltanBerrigomo 14 Jan 2016 4:18 UTC
0 points
in reply to: Silver_Swift’s comment on: What can go wrong with the following protocol for AI containment?
Here is my attempt at a calculation. Disclaimer: this is based on googling. If you are actually knowledgeable in the subject, please step in and set me right.

There are 10^11 neurons in the human brain.

A neuron will fire about 200 times per second.

It should take a constant number of flops to decide whether a neuron will fire —say 10 flops (no need to solve a differential equation, neural networks usually use some discrete heuristics for something like this)

I want a society of 10^6 orcs running for 10^6 years

As you suggest, lets let the simulation run for a year of real time (moving away at this point from my initial suggestion of 1 second). By my calculations, it seems that in order for this to happen we need a computer that does 2x10^25 flops per second.

According to this

http://www.datacenterknowledge.com/archives/2015/04/15/doe-taps-intel-cray-to-build-worlds-fastest-supercomputer/

...in 2018 we will have a supercomputer that does about 2x10^17 flops per second.

That means we need a computer that is one hundred million times faster than the best computer in 2018.

That is still quite a lot, of course. If Moore’s law was ongoing, this would take ~40 years; but Moore’s law is dying. Still, it is not outside the realm of possibility for, say, the next 100 years.

Edit: By the way, one does not need to literally implement what I suggested—the scheme I suggested is in principle applicable whenever you have a superintelligence, regardless of how it was designed.

Indeed, if we somehow develop an above-human intelligence, rather than trying to make sure its goals are aligned with ours, we might instead let it loose within a simulated world, giving it a preference for continued survival. Just one superintelligence thinking about factoring for a few thousand simulated years would likely be enough to let us factor any number we want. We could even give it have in-simulation ways of modifying its own code.

ZoltanBerrigomo 14 Jan 2016 4:01 UTC
0 points
in reply to: Slider’s comment on: What can go wrong with the following protocol for AI containment?

Still inability to realise what you are doing seems rather dangerous.

So far, all I’ve done is post a question on lesswrong :)

More seriously, I do regret it if I appeared unaware of the potential danger. I am of course aware of the possibility that experiments with AI might destroy humanity. Think of my post above a suggesting a possible approach to investigate—perhaps one with some kinks as written (that is why I’m asking a question here) but (I think) with the possibility of one day having rigorous safety guarantees.

ZoltanBerrigomo 14 Jan 2016 3:56 UTC
0 points
in reply to: Silver_Swift’s comment on: What can go wrong with the following protocol for AI containment?
I think this calculation too conservative. The reason is (as I understand it) that neurons are governed by various differential equations, and simulating them accurately is a pain in the ass. We should instead assume that deciding whether a neuron will fire will take a constant number of flops.

I’ll write another comment which attempts to redo your calculation with different assumptions.

It seems to me that by the time we can do that, we should have figured out a better way to create AI.

But will we have figured a way to reap the gains of AI safely for humanity?

ZoltanBerrigomo 14 Jan 2016 3:47 UTC
0 points
in reply to: Silver_Swift’s comment on: What can go wrong with the following protocol for AI containment?

but more than that, it knows that the real universe is capable of producing intelligent beings that chose this particular world to simulate.

Good point—this undermines a lot of what I wrote in my update 1. For example, I have no idea if F = m d^3 x / dt would result in a world that is capable of producing intelligent beings.

I should at some point produce a version of the above post with this claim, and other questionable parenthetical remarks I made, deleted, or at least acknowledging that they require further argumentation; they are not necessary for the larger point, which is that as long as the only thing the superintelligence can do (by definition) is live in a simulated world governed by Newton’s laws, and as long as we don’t interact with it at all except to see an automatically verified answer to a preset question (e.g., factor “111000232342342”), there is nothing it can do to harm us.

ZoltanBerrigomo 14 Jan 2016 3:41 UTC
0 points
in reply to: Lukas_Gloor’s comment on: What can go wrong with the following protocol for AI containment?
I guess I am willing to bite the bullet and say that, as long as entity X prefers existence to nonexistence, you have done it no harm by bringing it into being. I realize this generates a number of repulsive-sounding conclusions, e.g., it becomes ethical to create entities which will live, by our 21st century standards, horrific lives.

At least some of them will tell you they had rather not been born.

If one is willing to accept my reasoning above, I think one can take one more leap and say that statistically as long as the vast majority of these entities will prefer existing to never having been brought into being, we are in the clear.

ZoltanBerrigomo 14 Jan 2016 3:37 UTC
0 points
in reply to: HungryHobo’s comment on: What can go wrong with the following protocol for AI containment?
The theorem you cite (provided I understood you correctly) does not preclude the possibility of checking whether a program written in a certain pre-specified format will have bugs. Bugs here are defined to be certain undesirable properties (e.g., looping forever, entering certain enumerated states, etc).

Baby versions of such tools (which automatically check whether your program will have certain properties from inspecting the code) already exist.

ZoltanBerrigomo 12 Jan 2016 21:04 UTC
0 points
in reply to: HungryHobo’s comment on: What can go wrong with the following protocol for AI containment?
Nice idea...I wrote an update to the post suggesting what seemed to me to be a variation on your suggestion.

About program checking: I agree completely. I’m not very informed about the state of the art, but it is very plausible that what we know right now is not yet up to task.

ZoltanBerrigomo 12 Jan 2016 20:23 UTC
0 points
in reply to: Silver_Swift’s comment on: What can go wrong with the following protocol for AI containment?

I’m a bit skeptical about whether you can actually create a superintelligent AI by combining sped up humans like that,

Why not? You are pretty smart, and all you are is a combination of 10^11 or so very “dumb” neurons. Now imagine a “being” which is actually a very large number of human-level intelligences, all interacting...

ZoltanBerrigomo 12 Jan 2016 20:21 UTC
1 point
in reply to: Silver_Swift’s comment on: What can go wrong with the following protocol for AI containment?

is very, very difficult not to give a superintelligence any hints of how the physics of our world work.

I wrote a short update to the post which tries to answer this point.

Maybe they notice minor fluctuations in the speed of the simulation based on environmental changes to the hardware

I believe they should have no ability whatsoever to detect fluctuations in the speed of the simulation.

Consider how the world of world of warcraft appears to an orc inside the game. Can it tell the speed at which the hardware is running the game?

It can’t. What it can do is compare the speed of different things: how fast does an apple fall from a tree vs how fast a bird flies across the sky.

The orc’s inner perception of the flow of time is based on comparing these things (e.g., how fast does an apple fall) to how fast their simulated brains process information.

If everything is slowed down by a factor of 2 (so you, as a player, see everything twice is slow), nothing appears any different to a simulated being within the simulation.

ZoltanBerrigomo 12 Jan 2016 20:03 UTC
0 points
in reply to: Richard_Kennaway’s comment on: What can go wrong with the following protocol for AI containment?
These are good points. Perhaps I should not have said “interact” but chosen a different word instead. Still, its ability to play us is limited since (i) we will be examining the records of the world after it is dead (ii) it has no opportunity to learn anything about us.

Edit: we might even make it impossible for it to game us in the following way. All records of the simulated world are automatically deleted upon completion—except for a specific prime factorization we want to know.

This is a really bad argument for safety.

You are right, of course. But you wrote that in response to what was a parenthetical remark on my part—the real solution is to use program checking to make sure the laws of physics of the simulated world are never violated.

ZoltanBerrigomo 12 Jan 2016 5:59 UTC
0 points
in reply to: Manfred’s comment on: What can go wrong with the following protocol for AI containment?
1. When talking about dealing and (non)interacting with real AIs, one is always talking about a future world with significant technological advances relative to our world today.
2. If we can formulate something as a question about math, physics, chemistry, biology, then we can potentially attack it with this scheme. These are definitely problems we really want to solve.
3. Its true that if we allow AIs more knowledge and more access to our world, they could potentially help us more—but of course the number of things that can go wrong has to increase as well. Perhaps a compromise which sacrifices some of the potential while decreasing the possibilities that can go wrong is better.

What can go wrong with the following protocol for AI containment?

ZoltanBerrigomo11 Jan 2016 23:03 UTC

0 points

29 comments6 min readLW link

ZoltanBerrigomo 11 Jan 2016 3:34 UTC
2 points
in reply to: ChristianKl’s comment on: Why CFAR’s Mission?

1+1=2 is true by definition of what 2 means

Russell and Whitehead would beg to differ.

ZoltanBerrigomo 11 Jan 2016 3:28 UTC
1 point
in reply to: Kaj_Sotala’s comment on: Why CFAR’s Mission?

Sometimes you’re dealing with a domain where explicit reasoning provides the best evidence, sometimes with a domain where emotions provide the best evidence.

And how should you (rationally) decide which kind of domain you are in?

Answer: using reason, not emotions.

Example: if you notice that your emotions have been a good guide in understanding what other people are thinking in the past, you should trust them in the future. The decision to do this, however, is an application of inductive reasoning.

ZoltanBerrigomo 8 Jan 2016 23:55 UTC
0 points
in reply to: ChristianKl’s comment on: Why CFAR’s Mission?

No. CFAR rationality is about aligning system I and system II. It’s not about declaring system I outputs to be worthy of being ignored in favor of system II outputs.

I believe you are nitpicking here.

If your reason tells you 1+1=2 but your emotions tell you that 1+1=3, being rational means going with your reason. If your reason tells you that ghosts do not exist, you should believe this to be the case even if you really, really want there to be evidence of an afterlife.

CFAR may teach you techniques to align your emotions and reason, but this does not change the fundamental fact that being rational involves evaluating claims like “is 1+1=2?” or empirical facts about the world such as “is there evidence for the existence of ghosts?” based on reason alone.

Just to forestall the inevitable objections (which always come in droves whenever I argue with anyone on this site): this does not mean you don’t have emotions; it does not mean that your emotions don’t play a role in determining your values; it does not mean that you shouldn’t train your emotions to be an aid in your decision-making, etc etc etc.

ZoltanBerrigomo 5 Jan 2016 5:51 UTC
1 point
in reply to: ChristianKl’s comment on: Why CFAR’s Mission?
Sure, you can work towards feeling more strongly about something, but I don’t believe you’ll ever be able match the emotional fervor the partisans feel -- I mean here the people who stew in their anger and embrace their emotions without reservations.

As a (rather extreme) example, consider Hitler. He was able to sway a great many people with what were appeals to anger and emotion (though I acknowledge there is much more to the phenomena of Hitler than this). Hypothetically, if you were a politician from the same era, say a rational one, and you understood that the way to persuade people is to tap into the public’s sense of anger, I’m not sure you’d be able to match him.

ZoltanBerrigomo 3 Jan 2016 17:42 UTC
1 point
in reply to: ChristianKl’s comment on: Why CFAR’s Mission?

Do the extend that it does require luck that simply means that it’s important to have more people with rationality + competence + caring. If you have many people some will get lucky.

The “little bit of luck” in my post above was something of an understatement; actually, I’d suggest it requires a lot of luck (among many other things) to successfully change the world.

I think you might be pattern matching to straw-vulcan rationality, that’s distinct from what CFAR wants to teach.

Not sure if I am, but I believe I am making a correct claim about human psychology here.

Being rational means many things, but surely one of them is making decisions based on some kind of reasoning process as opposed to recourse to emotions.

This does not mean you don’t have emotions.

You might, for example, have very strong emotions about matters pertaining to fights between your perceived in-group and out-group, but you try to put those aside and make judgments based on some sort of fundamental principles.

Now if, in the real world, the way you persuade people is by emotional appeals (and this is at least partially true), this will be more difficult the more you get in the habit of rational thinking, even if you have an accurate model about what it takes to persuade someone -- emotions are not easy to fake and humans have strong intuitions about whether someone’s expressed feelings are genuine.

ZoltanBerrigomo 1 Jan 2016 21:10 UTC
4 points
on: Why CFAR’s Mission?
A very interesting and thought provoking post—I especially like the Q & A format.

I want to quibble with one bit:

How can I tell there aren’t enough people out there, instead of supposing that we haven’t yet figured out how to find and recruit them?

Basically, because it seems to me that if people had really huge amounts of epistemic rationality + competence + caring, they would already be impacting these problems. Their huge amounts of epistemic rationality and competence would allow them to find a path to high impact; and their caring would compel them to do it.

There is an empirical claim about the world that is implicit in that statement, and it is this claim I want to disagree with. Namely: I think having a high impact on the world is really, really hard. I would suggest it requires more than just rationality + competence + caring; for one thing, it requires a little bit of luck.

It also requires a good ability to persuade others who are not thinking rationally. Many such people respond to unreasonable confidence, emotional appeals, salesmanship, and other rhetorical tricks which may be more difficult to produce the more you are used to thinking things through rationally.

ZoltanBerrigomo 1 Nov 2015 19:02 UTC
1 point
on: Rationality Quotes Thread October 2015

For those people who insist, however, that the only thing that is important is that the theory agrees with experiment, I would like to make an imaginary discussion between a Mayan astronomer and his student...

These are the opening words of a ~1.5 minute monologue in one of Feynman’s lectures; I won’t transcribe the remainder but it can be viewed here.

ZoltanBerrigomo

What can go wrong with the fol­low­ing pro­to­col for AI con­tain­ment?

What can go wrong with the following protocol for AI containment?