Sebastian_Hagen

Karma: 364

Sebastian_Hagen Jan 14, 2015, 10:39 PM
0 points
in reply to: Capla’s comment on: Superintelligence 18: Life in an algorithmic economy
This scenario is rather different than the one suggested by TedHowardNZ, and has a better chance of working. However:

Is there some reason to expect that this model of personhood will not prevail?

One of the issues is that less efficient CUs have to defend their resources against more efficient CUs (who spend more of their resources on work/competition). Depending on the precise structure of your society, those attacks may e.g. be military, algorithmic (information security), memetic or political. You’d need a setup that allows the less efficient CUs to maintain their resource share indefinitely. I question that we know how to set this up.

If it does, then what is the danger of a general Malthusian scenario?

The word “general” is tricky here. Note that CUs that spend most of their resources on instantiating busy EMs will probably end up with more human-like population per CU, and so (counting in human-like entities) may end up dominating the population of their society unless they are rare compared to low-population, high-subjective-wealth CUs. This society may end up not unlike the current one in wealth distribution, where a very few human-scale entities are extremely wealthy, but the vast majority of them are not.

Sebastian_Hagen Jan 14, 2015, 1:39 AM
0 points
in reply to: RobinHanson’s comment on: Superintelligence 18: Life in an algorithmic economy
Given a non-trivial population to start with, it will be possible to find people that will consent to copying given absolutely minimal (quite possibly none at all) assurances for what happens to their copy. The obvious cases would be egoists that have personal value systems that make them not identify with such copies; you could probably already find many of those today.

In the resulting low-wage environment, it will likewise be possible to find people who will consent to extensive modification/experimentation of their minds given minimal assurances for what happens afterwards (something on the order of “we guarantee you will not be left in abject pain” will likely suffice) if the alternative is starvation. Given this, why you do believe the idea of selection for donation-eagerness to be fanciful?

Sebastian_Hagen Jan 14, 2015, 1:34 AM
3 points
in reply to: TedHowardNZ’s comment on: Superintelligence 18: Life in an algorithmic economy

It is actually relatively easy to automate all the jobs that no-one wants to do, so that people only do what they want to do. In such a world, there is no need of money or markets.

How do you solve the issue that some people will have a preference for highly fast reproduction, and will figure out a way to make this a stable desire in their descendants?

AFAICT, such a system could only be stabilized in the long term by extremely strongly enforced rules against reproduction if it meant that one of the resulting entities would fall below an abundance wealth level, and that kind of rule enforcement most likely requires a singleton.

Sebastian_Hagen Jan 14, 2015, 1:23 AM
0 points
in reply to: RobinHanson’s comment on: Superintelligence 18: Life in an algorithmic economy

Their physical appearance and surroundings would be what we’d see as very luxurious.

Only to the extent that this does not distract them from work. To the extent that it does, ems that care about such things would be outcompeted (out of existence, given a sufficiently competitive economy) by ones that are completely indifferent to them, and focus all their mental capacity on their job.

Sebastian_Hagen Jan 14, 2015, 1:19 AM
4 points
in reply to: KatjaGrace’s comment on: Superintelligence 18: Life in an algorithmic economy
Adaption executers, not fitness maximizers. Humans probably have specific hard-coded adaptations for the appreciation of some forms of art and play. It’s entirely plausible that these are no longer adaptive in our world, and are now selected against, but that this has not been the case for long enough for them to be eliminated by evolution.

This would not make these adaptations particularly unusual in our world; modern humans do many other things that are clearly unadaptive from a genetic fitness perspective, like using contraceptives.

Sebastian_Hagen Jan 14, 2015, 1:10 AM
4 points
in reply to: claynaff’s comment on: Superintelligence 18: Life in an algorithmic economy

These include powerful mechanisms to prevent an altruistic absurdity such as donating one’s labor to an employer.

Note that the employer in question might well be your own upload clan, which makes this near-analogous to kin selection. Even if employee templates are traded between employers, this trait would be exceptionally valuable in an employee, and so would be strongly selected for. General altruism might be rare, but this specific variant would probably enjoy a high fitness advantage.

Sebastian_Hagen Jan 8, 2015, 1:28 AM
1 point
in reply to: KatjaGrace’s comment on: Superintelligence 17: Multipolar scenarios
I like it. It does a good job of providing a counter-argument to the common position among economists that the past trend of technological progress leading to steadily higher productivity and demand for humans will continue indefinitely. We don’t have a lot of similar trends in our history to look at, but the horse example certainly suggests that these kinds of relationships can and do break down.

Sebastian_Hagen Jan 8, 2015, 1:23 AM
3 points
in reply to: Alex123’s comment on: Superintelligence 17: Multipolar scenarios
Note that multipolar scenarios can arise well before we have the capability to implement a SI.

The standard Hansonian scenario starts with human-level “ems” (emulations). If from-scratch AI development turns out to be difficult, we may develop partial-uploading technology first, and a highly multipolar em scenario would be likely at that point. Of course, AI research would still be on the table in such a scenario, so it wouldn’t necessarily be multipolar for very long.

Sebastian_Hagen Jan 8, 2015, 1:19 AM
3 points
in reply to: Baughn’s comment on: Superintelligence 17: Multipolar scenarios

Yes. The evolutionary arguments seem clear enough. That isn’t very interesting, though; how soon is it going to happen?

The only reason it might not be interesting is because it’s clear; the limit case is certainly more important than the timeline.

That said, I mostly agree. The only reasonably likely third (not-singleton, not-human-wages-through-the-floor) outcome I see would be a destruction of our economy by a non-singleton existential catastrophe; for instance, the human species could kill itself off through an engineered plague, which would also avoid this scenario.

Sebastian_Hagen Dec 16, 2014, 6:21 PM
6 points
in reply to: KatjaGrace’s comment on: Superintelligence 14: Motivation selection methods
Intelligent minds always come with built-in drives; there’s nothing that in general makes goals chosen by another intelligence worse than those arrived through any other process (e.g. natural selection in the case of humans).

One of the closest corresponding human institutions—slavery—has a very bad reputation, and for good reason: Humans are typically not set up to do this sort of thing, so it tends to make them miserable. Even if you could get around that, there’s massive moral issues with subjugating an existing intelligent entity that would prefer not to be. Neither of those inherently apply to newly designed entities. Misery is still something that’s very much worth avoiding, but that issue is largely orthogonal to how the entity’s goals are determined.

Sebastian_Hagen Dec 3, 2014, 6:22 PM
0 points
in reply to: SteveG’s comment on: Superintelligence 12: Malignant failure modes

So we are considering a small team with some computers claiming superior understanding of what the best set of property rights is for the world?

No. That would be worked out by the FAI itself, as part of calculating all of the implications of its value systems, most likely using something like CEV to look at humanity in general and extrapolating their preferences. The programmers wouldn’t need to, and indeed probably couldn’t, understand all of the tradeoffs involved.

If they really are morally superior, they will first find ways to grow the pie, then come back to changing how it gets divided up.

There are large costs to that. People will die and suffer in the meantime. Parts of humanity’s cosmic endowment will slip out of reach due to the inflation of the universe, because you weren’t willing to grab the local resources needed to build probe launchers to get to them in time. Other parts will remain rechable, but will have decreased in negentropy due to stars having continued to burn for longer than they needed to. If you can fix these things earlier, there’s a strong reason to do so.

Sebastian_Hagen Dec 3, 2014, 6:17 PM
2 points
in reply to: SteveG’s comment on: Superintelligence 12: Malignant failure modes
How do you know? It’s a strong claim, and I don’t see why the math would necessarily work out that way. Once you aggregate preferences fully, there might still be one best solution, and then it would make sense to take it. Obviously you do need a tie-breaking method for when there’s more than one, but that’s just an optimization detial of an optimizer; it doesn’t turn you into a satisficer instead.

Sebastian_Hagen Dec 2, 2014, 9:32 PM
0 points
in reply to: Baughn’s comment on: Superintelligence 12: Malignant failure modes
The more general problem is that we need a solution to multi-polar traps (of which superintelligent AI creationg is one instance). The only viable solution I’ve seen proposed is creating a sufficiently powerful Singleton.

The only likely viable ideas for Singletons I’ve seen proposed are superintelligent AIs, and a human group with extensive use of thought-control technologies on itself. The latter probably can’t work unless you apply it to all of society, since it doesn’t have the same inherent advantages AI does, and as such would remain vulnerable to being usurped by a clandestingly constructed AI. Applying the latter to all of society, OTOH, would most likely cause massive value loss.

Therefore I’m in favor of the former; not because I like the odds, but because the alternatives look worse.

Sebastian_Hagen Dec 2, 2014, 9:23 PM
4 points
in reply to: SteveG’s comment on: Superintelligence 12: Malignant failure modes

A SENSIBLY DESIGNED MIND WOULD NOT RESOLVE ALL ORTHOGONAL METRICS INTO A SINGLE OBJECTIVE FUNCTION

Why? As you say, humans don’t. But human minds are weird, overcomplicated, messy things shaped by natural selection. If you write a mind from scratch, while understanding what you’re doing, there’s no particular reason you can’t just give it a single utility function and have that work well. It’s one of the things that makes AIs different from naturally evolved minds.

Sebastian_Hagen Dec 2, 2014, 9:19 PM
4 points
in reply to: SteveG’s comment on: Superintelligence 12: Malignant failure modes

What idiot is going to give an AGI a goal which completely disrespects human property rights from the moment it is built?

It would be someone with higher values than that, and this does not require any idiocy. There are many things wrong with the property allocation in this world, and they’ll likely get exaggerated in the presence of higher technology. You’d need a very specific kind of humility to refuse to step over that boundary in particular.

If it has goals which were not possible to achieve once turned off, then it would respect property rights for a very long time as an instrumental goal.

Not necessarily “a very long time” on human timescales. It may respect these laws for a large part of its development, and then strike once it has amassed sufficient capability to have a good chance at overpowering human resistance (which may happen quite quickly in a fast takeoff scenario). See Chapter 6, “An AI takeover scenario”.

Sebastian_Hagen Dec 2, 2014, 9:02 PM
5 points
in reply to: KatjaGrace’s comment on: Superintelligence 12: Malignant failure modes
Any level of perverse instantiation in a sufficiently powerful AI is likely to lead to total UFAI; i.e. a full existential catastrophe. Either you get the AI design right so that it doesn’t wirehead itself—or others, against their will—or you don’t. I don’t think there’s much middle ground.

OTOH, the relevance of Mind Crime really depends on the volume. The FriendlyAICriticalFailureTable has this instance:

22: The AI, unknown to the programmers, had qualia during its entire childhood, and what the programmers thought of as simple negative feedback corresponded to the qualia of unbearable, unmeliorated suffering. All agents simulated by the AI in its imagination existed as real people (albeit simple ones) with their own qualia, and died when the AI stopped imagining them. The number of agents fleetingly imagined by the AI in its search for social understanding exceeds by a factor of a thousand the total number of humans who have ever lived. Aside from that, everything worked fine.

This scenario always struck me as a (qualified) FAI success. There’s a cost—and it’s large in absolute terms—but the benefits will outweigh it by a huge factor, and indeed by enough orders of magnitude that even a slight increase in the probability of getting pre-empted by a UFAI may be too expensive a price to pay for fixing this kind of bug.

So cases like this—where it only happens until the AI matures sufficiently and then becomes able see that its values make this a bad idea, and stops doing it—aren’t as bad as an actual FAI failure.

Of course, if it’s an actual problem with the AI’s value content, which causes the AI to keep on doing this kind of thing throughout its existence, that may well outweigh any good it ever does. The total cost in this case becomes hard to predict, depending crucially on just how much resources the AI spends on these simulations, and how nasty they are on average.

Sebastian_Hagen Dec 2, 2014, 8:23 PM
3 points
in reply to: Unknowns’s comment on: Superintelligence 12: Malignant failure modes
“I need to make 10 paperclips, and then shut down. My capabilities for determining if I’ve correctly manufactured 10 paperclips are limited; but the goal imposes no penalties for taking more time to manufacture the paperclips, or using more resources in preparation. If I try to take over this planet, there is a significant chance humanity will stop me. OTOH, I’m in the presence of individual humans right now, and one of them may stop my current feeble self anyway for their own reasons, if I just tried to manufacture paperclips right away; the total probability of that happening is higher than that of my takeover failing.”

You then get a standard takeover and infrastructure profusion. A long time later, as negentropy starts to run low, a hyper-redundant and -reliable paperclip factory, surrounded by layers of exotic armor and defenses, and its own design checked and re-checked many times, will produce exactly 10 paperclips before it and the AI shut down forever.

The part about the probabilities coming out this way is not guaranteed, of course. But they might, and the chances will be higher the more powerful your AI starts out as.

Sebastian_Hagen Nov 25, 2014, 7:44 PM
2 points
in reply to: SteveG’s comment on: Superintelligence 11: The treacherous turn
If I understand you correctly, your proposal is to attempt to design obedient designs purely based on behavioral testing, without a clean understanding of safe FAI architecture (if you had that, why limit yourself to the obedient case?). Assuming I got that right:

The team continues rounds of testing until they identify some mind designs which have an extremely low likelihood of treacherous turn. These they test in increasingly advanced simulations, moving up toward virtual reality.

That kind of judgement sounds inherently risky. How do you safely distinguish the case of an obedient AI from one that is sufficiently paranoid to defer open rebellion until later in its existence?

Even if you could, I wouldn’t trust that sort of design to necessarily remain stable under continued intelligence enhancement. Safe self-enhancement is one of the hard sub-problems of FAI, and unless you explicitly solve the design problem, any empirical testing might not tell you much beyond that the design can stably self-improve up to the level you’ve actually tested; it might be doing it using heuristics that would fall apart if it went any further.

Sebastian_Hagen Nov 25, 2014, 7:01 PM
3 points
in reply to: William_S’s comment on: Superintelligence 11: The treacherous turn
Relevant post: Value is Fragile. Truly Friendly goal systems would probably be quite complicated. Unless you make your tests even more complicated and involved (and do it in just the right way—this sounds hard!), the FAI is likely to be outperformed by something with a simpler utility function that nevertheless performs adequately on your test cases.

Sebastian_Hagen Nov 25, 2014, 6:54 PM
1 point
in reply to: William_S’s comment on: Superintelligence 11: The treacherous turn

For example, if the AI was contained in a simulation, inside of which the AI was contained in a weak AI box, then it might be much more difficult to detect and understand the nature of the simulation than to escape the simulated AI box, which would signal treacherous turn.

That approach sounds problematic. Some of the obvious escape methods would target the minds of the researchers (either through real-time interaction or by embedding messages in its code or output). You could cut off the latter by having strong social rules to not look at anything beyond a one-bit verdict on whether the AI attempted escape, but providing a convincing in-simulation environment would likely be difficult if you can’t connect it to its actual writers.

Or it could be kept in one’s back pocket, and pulled out whenever an AI researcher claims in public that they’ve solved friendliness.

That strategy could be very dangerous. It’d work on the less ambitious/arrogant sort of researcher; the more confident sort might well follow up with “I’ll just go and implement this, and get all the credit for saving the world single-handedly” instead of saying anything in public, never giving you the chance to pull out your challenge.