Sebastian_Hagen

Karma: 363

Sebastian_Hagen 17 Sep 2015 20:12 UTC
2 points
in reply to: DataPacRat’s comment on: Probabilities Small Enough To Ignore: An attack on Pascal’s Mugging

In parallel, if I am to compare two independent scenarios, the at-least-one-in-ten-billion odds that I’m hallucinating all this, and the darned-near-zero odds of a Pascal’s Mugging attempt, then I should be spending proportionately that much more time dealing with the Matrix scenario than that the Pascal’s Mugging attempt is true

That still sounds wrong. You appear to be deciding on what to precompute for purely by probability, without considering that some possible futures will give you the chance to shift more utility around.

If I don’t know anything about Newcomb’s problem and estimate a 10% chance of Omega showing up and posing it to me tomorrow, I’ll definitely spend more than 10% of my planning time for tomorrow reading up on and thinking about it. Why? Because I’ll be able to make far more money in that possible future than the others, which means that the expected utility differentials are larger, and so it makes sense to spend more resources on preparing for it.

The I-am-undetectably-insane case is the opposite of this, a scenario that it’s pretty much impossible to usefully prepare for.

And a PM scenario is (at least for an expected-utility maximizer) a more extreme variant of my first scenario—low probabilities of ridiculously large outcomes, that are because of that still worth thinking about.

Sebastian_Hagen 17 Sep 2015 13:59 UTC
7 points
in reply to: ike’s comment on: Probabilities Small Enough To Ignore: An attack on Pascal’s Mugging
Continuity and independence.

Continuity: Consider the scenario where each of the [LMN] bets refer to one (guaranteed) outcome, which we’ll also call L, M and N for simplicity.

Let U(L) = 0, U(M) = 1, U(N) = 10**100

For a simple EU maximizer, you can then satisfy continuity by picking p=(1-1/10**100). A PESTI agent, OTOH, may just discard a (1-p) of 1/10**100, which leaves no other options to satisfy it.

The 10**100 value is chosen without loss of generality. For PESTI agents that still track probabilities of this magnitude, increase it until they don’t.

Independence: Set p to a number small enough that it’s Small Enough To Ignore. At that point, the terms for getting L and M by that probability become zero, and you get equality between both sides.

Sebastian_Hagen 17 Sep 2015 13:12 UTC
6 points
in reply to: DataPacRat’s comment on: Probabilities Small Enough To Ignore: An attack on Pascal’s Mugging

Thus, I can never be more than one minus one-in-ten-billion sure that my sensory experience is even roughly correlated with reality. Thus, it would require extraordinary circumstances for me to have any reason to worry about any probability of less than one-in-ten-billion magnitude.

No. The reason not to spend much time thinking about the I-am-undetectably-insane scenario is not, in general, that it’s extraordinarily unlikely. The reason is that you can’t make good predictions about what would be good choices for you in worlds where you’re insane and totally unable to tell.

This holds even if the probability for the scenario goes up.

Sebastian_Hagen 1 May 2015 22:30 UTC
1 point
on: Meetup : Dublin
I’ll be there.

Sebastian_Hagen 31 Mar 2015 21:25 UTC
6 points
in reply to: KatjaGrace’s comment on: Superintelligence 29: Crunch time
It’s the most important problem of this time period, and likely human civilization as a whole. I donate a fraction of my income to MIRI.

Sebastian_Hagen 17 Mar 2015 19:16 UTC
1 point
in reply to: KatjaGrace’s comment on: Superintelligence 27: Pathways and enablers

Which means that if we buy this [great filter derivation] argument, we should put a lot more weight on the category of ‘everything else’, and especially the bits of it that come before AI. To the extent that known risks like biotechnology and ecological destruction don’t seem plausible, we should more fear unknown unknowns that we aren’t even preparing for.

True in principle. I do think that the known risks don’t cut it; some of them might be fairly deadly, but even in aggregate they don’t look nearly deadly enough to contribute much to the great filter. Given the uncertainties in the great filter analysis, that conclusion for me mostly feeds back in that direction, increasing the probability that the GF is in fact behind us.

Your SIA doomsday argument—as pointed out by michael vassar in the comments—has interesting interactions with the simulation hypothesis; specifically, since we don’t know if we’re in a simulation, the bayesian update in step 3 can’t be performed as confidently as you stated. Given this, “we really can’t see a plausible great filter coming up early enough to prevent us from hitting superintelligence” is also evidence for this environment being a simulation.

Sebastian_Hagen 4 Mar 2015 1:53 UTC
1 point
in reply to: KatjaGrace’s comment on: Superintelligence 25: Components list for acquiring values
This issue is complicated by the fact that we don’t really know how much computation our physics will give us access to, or how relevant negentropy is going to be in the long run. In particular, our physics may allow access to (countably or more) infinite computational and storage resources given some superintelligent physics research.

For Expected Utility calculations, this possibility raises the usual issues of evaluating potential infinite utilities. Regardless of how exactly one decides to deal with those issues, the existence of this possibility does shift things in favor of prioritizing for safety over speed.

Sebastian_Hagen 21 Feb 2015 19:01 UTC
1 point
in reply to: PhilGoetz’s comment on: Superintelligence 23: Coherent extrapolated volition
I used “invariant” here to mean “moral claim that will hold for all successor moralities”.

A vastly simplified example: at t=0, morality is completely undefined. At t=1, people decide that death is bad, and lock this in indefinitely. At t=2, people decide that pleasure is good, and lock that in indefinitely. Etc.

An agent operating in a society that develops morality like that, looking back, would want to have all the accidents that lead to current morality to be maintained, but looking forward may not particularly care about how the remaining free choices come out. CEV in that kind of environment can work just fine, and someone implementing it in that situation would want to target it specifically at people from their own time period.

Sebastian_Hagen 17 Feb 2015 22:59 UTC
3 points
in reply to: PhilGoetz’s comment on: Superintelligence 23: Coherent extrapolated volition
That does not sound like much of a win. Present-day humans are really not that impressive, compared to the kind of transhumanity we could develop into. I don’t think trying to reproduce entites close to our current mentality is worth doing, in the long run.

Sebastian_Hagen 17 Feb 2015 22:49 UTC
5 points
in reply to: Furcas’s comment on: Superintelligence 23: Coherent extrapolated volition
While that was phrased in a provocative manner, there /is/ an important point here: If one has irreconcilable value differences with other humans, the obvious reaction is to fight about them; in this case, by competing to see who can build an SI implementing theirs first.

I very much hope it won’t come to that, in particular because that kind of technology race would significantly decrease the chance that the winning design is any kind of FAI.

In principle, some kinds of agents could still coordinate to avoid the costs of that kind of outcome. In practice, our species does not seem to be capable of coordination at that level, and it seems unlikely that this will change pre-SI.

Sebastian_Hagen 17 Feb 2015 22:40 UTC
1 point
in reply to: Richard_Kennaway’s comment on: Superintelligence 23: Coherent extrapolated volition
True, but it would nevertheless make for a decent compromise. Do you have a better suggestion?

Sebastian_Hagen 17 Feb 2015 22:39 UTC
1 point
in reply to: diegocaleiro’s comment on: Superintelligence 23: Coherent extrapolated volition

allocating some defense army patrol keeping the borders from future war?

Rather than use traditional army methods, it’s probably more efficient to have the SI play the role of Sysop in this scenario, and just deny human actors access to base-layer reality; though if one wanted to allow communication between the different domains, the sysop may still need to run some active defense against high-level information attacks.

Sebastian_Hagen 17 Feb 2015 22:26 UTC
3 points
in reply to: PhilGoetz’s comment on: Superintelligence 23: Coherent extrapolated volition
That seems wrong.

As a counterexample, consider a hypothetical morality development model where as history advances, human morality keeps accumulating invariants, in a largely unpredictable (chaotic) fashion. In that case modern morality would have more invariants than that of earlier generations. You could implement a CEV from any time period, but earlier time periods would lead to some consequences that by present standards are very bad, and would predictably remain very bad in the future; nevertheless, a present-humans CEV would still work just fine.

Sebastian_Hagen 3 Feb 2015 20:55 UTC
3 points
in reply to: KatjaGrace’s comment on: Superintelligence 21: Value learning
Perhaps. But it is a desperate move, both in terms of predictability and in terms of the likely mind crime that would result in its implementation, since the conceptually easiest and most accurate ways to model other civilizations would involve fully simulating the minds of their members.

If we had to do it, I would be much more interested in aiming it at slightly modified versions of humanity as opposed to utterly alien civilizations. If everyone in our civilization had taken AI safety more seriously, and we could have coordinated to wait a few hundred years to work out the issues before building one, what kind of AI would our civilization have produced? I suspect the major issue with this approach is formalizing “If everyone in our civilization had taken AI safety more seriously” for the purpose of aiming an HM-implementing AI at those possibilities in particular.

Sebastian_Hagen 3 Feb 2015 20:49 UTC
3 points
in reply to: KatjaGrace’s comment on: Superintelligence 21: Value learning
I agree, the actual local existence of other AIs shouldn’t make a difference, and the approach could work equally either way. As Bostrom says on page 198, no communication is required.

Nevertheless, for the process to yield a useful result, some possible civilization would have to build a non-HM AI. That civilization might be (locally speaking) hypothetical or simulated, but either way the HM-implementing AI needs to think of it to delegate values. I believe that’s what footnote 25 gets at: From a superrational point of view, if every possible civilization (or every one imaginable to the AI we build) at this point in time chooses to use an HM approach to value coding, it can’t work.

Sebastian_Hagen 3 Feb 2015 20:42 UTC
3 points
in reply to: KatjaGrace’s comment on: Superintelligence 21: Value learning
Powerful AIs are probably much more aware of their long-term goals and able to formalize them than a heterogenous civilization is. Deriving a comprehensive morality for post-humanity is really hard, and indeed CEV is designed to avoid the need of having humans do that. Doing it for an arbitrary alien civilization would likely not be any simpler.

Whereas with powerful AIs, you can just ask them which values they would like implemented and probably get a good answer, as proposed by Bostrom.

Sebastian_Hagen 3 Feb 2015 20:34 UTC
4 points
in reply to: KatjaGrace’s comment on: Superintelligence 21: Value learning
The Hail Mary and Christiano’s proposals, simply for not having read about them before.

Sebastian_Hagen 3 Feb 2015 0:39 UTC
0 points
in reply to: KatjaGrace’s comment on: Superintelligence 20: The value-loading problem
Davis massively underestimates the magnitude and importance of the moral questions we haven’t considered, which renders his approach unworkable.

I feel safer in the hands of a superintelligence who is guided by 2014 morality, or for that matter by 1700 morality, than in the hands of one that decides to consider the question for itself.

I don’t. Building a transhuman civilization is going to raise all sorts of issues that we haven’t worked out, and do so quickly. A large part of the possible benefits are going to be contingent on the controlling system becoming much better at answering moral questions than any individual humans are right now. I would be extremely surprised if we don’t end up losing at least one order of magnitude of utility to this approach, and it wouldn’t surprise me at all if it turns out to produce a hellish environment in short order. The cost is too high.

The superintelligence might rationally decide, like the King of Brobdingnag, that we humans are “the most pernicious race of little odious vermin that nature ever suffered to crawl upon the surface of the earth,” and that it would do well to exterminate us and replace us with some much more worthy species. However wise this decision, and however strongly dictated by the ultimate true theory of morality, I think we are entitled to object to it, and to do our best to prevent it.

I don’t understand what scenario he is envisioning, here. If (given sufficient additional information, intelligence, rationality and development time) we’d agree with the morality of this result, then his final statement doesn’t follow. If we wouldn’t, it’s a good old-fashioned Friendliness failure.

Sebastian_Hagen 3 Feb 2015 0:31 UTC
1 point
in reply to: ahbwramc’s comment on: Superintelligence 20: The value-loading problem

One obvious failure mode would be in specifying which dead people count—if you say “the people described in these books,” the AI could just grab the books and rewrite them. Hmm, come to think of it: is any attempt to pin down human preferences by physical reference rather than logical reference vulnerable to tampering of this kind, and therefore unworkable?

Not as such, no. It’s a possible failure mode, similar to wireheading; but both of those are avoidable. You need to write the goal system in such a way that makes the AI care about the original referent, not any proxy that it looks at, but there’s no particular reason to think that’s impossible.

In general though, I’m continually astounded at how many people, upon being introduced to the value loading problem and some of the pitfalls that “common-sense” approaches have, still say “Okay, but why couldn’t we just do [idea I came up with in five seconds]?”

Agreed.

Sebastian_Hagen 17 Jan 2015 22:38 UTC
0 points
in reply to: Capla’s comment on: Superintelligence 18: Life in an algorithmic economy
To the extent that CUs are made up of human-like entities (as opposed to e.g. more flexible intelligences that can scale to effectively use all their resources), one of the choices they need to make is how large an internal population to keep, where higher populations imply less resources per person (since the amount of resources per CU is constant).

Therefore, unless the high-internal-population CUs are rare, most of the human-level population will be in them, and won’t have resources of the same level as the smaller numbers of people in low-population CUs.