sbenthall

• sbenthallOct 20, 2014, 5:23 AM

  2 points

  in reply to: lukeprog’s comment on: What is op­ti­miza­tion power, for­mally?

  Thanks. That’s very helpful.

  I’ve been thinking about Stuart Russell lately, which reminds me...bounded rationality. Isn’t there a bunch of literature on that?

  http://​​en.wikipedia.org/​​wiki/​​Bounded_rationality

  Have you ever looked into any connections there? Any luck with that?

• sbenthallOct 20, 2014, 5:20 AM

  2 points

  in reply to: Gunnar_Zarncke’s comment on: What is op­ti­miza­tion power, for­mally?

  1) This is an interesting approach. It looks very similar to the approach taken by the mid-20th century cybernetics movement—namely, modeling social and cognitive feedback processes with the metaphors of electrical engineering. Based on this response, you in particular might be interested in the history of that intellectual movement.

  My problem with this approach is that it considers the optimization process as a black box. That seems particularly unhelpful when we are talking about the optimization process acting on itself as a cognitive process. It’s easy to imagine that such a thing could just turn itself into a superoptimizer, but that would not be taking into account what we know about computational complexity.

  I think that it’s this kind of metaphor that is responsible for “foom” intuitions, but I think those are misplaced.

  2) Partial differential equations assume continuous functions, no? But in computation, we are dealing almost always with discrete math. What do you think about using concepts from combinatorial optimization theory, since those are already designed to deal with things like optimization resources and optimization efficiency?

• sbenthallOct 20, 2014, 5:04 AM

  2 points

  on: Fix­ing Mo­ral Hazards In Busi­ness Science

  Could you please link to examples of the kind of marketing studies that you are talking about? I’d especially like to see examples of those that you consider good vs. those you consider bad.

• sbenthallOct 20, 2014, 5:03 AM

  3 points

  in reply to: DavidLS’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  I am confused. Shouldn’t the questions depend on the content of the study being performed? Which would depend (very specifically) on the users/​clients? Or am I missing something?

• sbenthallOct 20, 2014, 4:55 AM

  4 points

  in reply to: DavidLS’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  I would worry about sampling bias due to selection based on, say, enjoying points.

• sbenthallOct 20, 2014, 4:52 AM

  6 points

  in reply to: DavidLS’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  The privacy issue here is interesting.

  It makes sense to guarantee anonymity. Participants recruited personally by company founders may be otherwise unwilling to report honestly (for example). For health related studies, privacy is an issue for insurance reasons, etc.

  However, for follow-up studies, it seems important to keep earlier records including personally identifiable information so as to prevent repeatedly sampling from the same population.

  That would imply that your organization/​system needs to have a data management system for securely storing the personal data while making it available in an anonymized form.

  However, there are privacy risks associated with ‘anonymized’ data as well, since this data can sometimes be linked with other data sources to make inferences about participants. (For example, if participants provide a zip code and certain demographic information, that may be enough to narrow it down to a very few people.) You may want to consider differential privacy solutions or other kinds of data perturbation.

  http://​​en.wikipedia.org/​​wiki/​​Differential_privacy

• sbenthallOct 20, 2014, 4:39 AM

  5 points

  in reply to: DavidLS’s comment on: Fix­ing Mo­ral Hazards In Busi­ness Science

  He then takes whatever steps we decide on to locate participants.

  Even if the group assignments are random, the prior step of participant sampling could lead to distorted effects. For example, the participants could be just the friends of the person who created the study who are willing to shill for it.

  The studies would be more robust if your organization took on the responsibility of sampling itself. There is non-trivial scientific literature on the benefits and problems of using, for example, Mechanical Turk and Facebook ads for this kind of work. There is extra value added for the user/​client here, which is that the participant sampling becomes a form of advertising.

What is op­ti­miza­tion power, for­mally?

• sbenthallSep 24, 2014, 10:53 PM

  3 points

  in reply to: Randaly’s comment on: Depth-based su­per­con­trol­ler ob­jec­tives, take 2

  Thanks for your thoughtful response. I’m glad that I’ve been more comprehensible this time. Let me see if I can address the problems you raise:

  1) Point taken that human freedom is important. In the background of my argument is a theory that human freedom has to do with the endogeneity of our own computational process. So, my intuitions about the role of efficiency and freedom are different from yours. One way of describing what I’m doing is trying to come up with a function that a supercontroller would use if it were to try to maximize human freedom. The idea is that choices humans make are some of the most computationally complex things they do, and so the representations created by choices are deeper than others. I realize now I haven’t said any of that explicitly let alone argued for it. Perhaps that’s something I should try to bring up in another post.

  2) I also disagree with the morality of this outcome. But I suppose that would be taken as beside the point. Let me see if I understand the argument correctly: if the most ethical outcome is in fact something very simple or low-depth, then this supercontroller wouldn’t be able to hit that mark? I think this is a problem whenever morality (CEV, say) is a process that halts.

  I wonder if there is a way to modify what I’ve proposed to select for moral processes as opposed to other generic computational processes.

  3) A couple responses:

  • Oh, if you can just program in “keep humanity alive” then that’s pretty simple and maybe this whole derivation is unnecessary. But I’m concerned about the feasibility of formally specifying what is essential about humanity. VAuroch has commented that he thinks that coming up with the specification is the hard part. I’m trying to defer the problem to a simpler one of just describing everything we can think of that might be relevant. So, it’s meant to be an improvement over programming in “keep humanity alive” in terms of its feasibility, since it doesn’t require solving perhaps impossible problems of understanding human essence.

  • Is it the consensus of this community that finding an objective function in E is an easy problem? I got the sense from Bostrom’s book talk that existential catastrophe was on the table as a real possibility.

  I encourage you to read the original Bennett paper if this interests you. I think your intuitions are on point and appreciate your feedback.

• sbenthallSep 24, 2014, 10:32 PM

  1 point

  in reply to: VAuroch’s comment on: Depth-based su­per­con­trol­ler ob­jec­tives, take 2

  I see, that’s interesting. So you are saying that while the problem as scoped in §2 may take a function of arbitrary complexity, there is a constraint in the superintelligence problem I have missed, which is that the complexity of the objective function has certain computational limits.

  I think this is only as extreme a problem as you say in a hard takeoff situation. In a slower takeoff situation, inaccuracies due to missing information could be corrected on-line as computational capacity grows. This is roughly business-as-usual for humanity—powerful entities direct the world according to their current best theories; these are sometimes corrected.

  It’s interesting that you are arguing that if we knew what information to include in a full specification of humanity, we’d be making substantial progress towards the value problem. In §3.2 I argued that the value problem need only be solved with a subset of the full specification of humanity. The fullness of that specification was desirable just because it makes it less likely that we’ll be missing the parts that are important to value.

  If, on the other hand, that you are right and the full specification of humanity is important to solving the value problem—something I’m secretly very sympathetic to—then

  (a) we need a supercomputer capable of processing the full specification in order to solve the value problem, so unless there is an iterative solution here the problem is futile and we should just accept that The End Is Nigh, or else try, as I’ve done, to get something Close Enough and hope for slow takeoff, and

  (b) the solution to the value problem is going to be somewhere done the computational path from h and is exactly the sort of thing that would be covered in the scope of g*.

  It would be a very nice result, I think, if the indirect normativity problem or CEV or whatever could be expressed in terms of the the depth of computational paths from the present state of humanity for precisely this reason. I don’t think I’ve hit that yet exactly but it’s roughly what I’m going for. I think it may hinge on whether the solution to the value problem is something that involves a halting process, or whether really it’s just to ensure the continuation of human life (i.e. as a computational process). In the latter case, I think the solution is very close to what I’ve been proposing.

• sbenthallSep 24, 2014, 10:09 PM

  1 point

  in reply to: cousin_it’s comment on: Depth-based su­per­con­trol­ler ob­jec­tives, take 2

  Could you flesh this out? I’m not familiar with key-stretching.

  A pretty critical point is whether or not the hashed value is algorithmically random. The depth measure has the advantage of picking over all permissible starting conditions without having to run through each one. So it’s not exactly analogous to a brute force attack. So for the moment I’m not convinced on this argument.

• sbenthallSep 24, 2014, 1:54 AM

  2 points

  in reply to: Toggle’s comment on: Depth-based su­per­con­trol­ler ob­jec­tives, take 2

  Thanks for your encouraging comments. They are much appreciated! I was concerned that following the last post with an improvement on it would be seen as redundant, so I’m glad that this process has your approval.

  Regarding your first point:

  • Entropy is not depth. If you do something that increases entropy, then you actually reduce depth, because it is easier to get to what you have from an incompressible starting representation. In particular, the incompressible representation that matches the high-entropy representation you have created. So if you hold humanity steady and superheat the moon, you more or less just keep things at D(u) = D(h), with low D(u/​h).

  • You can do better if you freeze humanity and then create fractal grey goo, which is still in the spirit of your objection. Then you have high D(u), D(u/​h) is something like D(u) - D(h) except for when the fractal starts to reproduce human patterns out of the sheer vigor of its complexity, in which case I guess D(u/​h) would begin to drop...though I’m not sure. This may require a more thorough look at the mathematics. What do you think?

  Regarding your second point...

  Strictly speaking, I’m not requiring that h abstract away the fleshy bits and capture what is essentially human or transhuman. I am trying to make the objective function agnostic to these questions. Rather, h can include fleshy bits and all. What’s important is that it includes at least what is valuable, and that can be done by including anything that might be valuable. The needle in the haystack can be discovered later, if it’s there at all. Personally, I’m not a transhumanist. I’m an existentialist; I believe our existence precedes our essence.

  That said I think this is a clever point with substance to it. I am, in fact, trying to shift our problem-solving attention to other problems. However, I am trying to turn attention to more tractable and practical questions.

  One simple one is: how can we make better libraries for capturing human existence, so that a supercontroller could make use of as much data as possible as it proceeds?

  Another is: given that the proposed objective function is in fact impossible to compute, but (if the argument is ultimately successful) also given that it points in the right direction, what kinds of processes/​architectures/​algorithms would approximate a g-maximizing supercontroller? Since we have time to steer in the right direction now, how should we go about it?

  My real agenda is that I think that there are a lot of pressing practical questions regarding machine intelligence and its role in the world, and that the “superintelligence” problem is a distraction except that it can provide clearer guidelines of how we should be acting now.

• sbenthallSep 24, 2014, 1:29 AM

  2 points

  in reply to: jimrandomh’s comment on: Depth-based su­per­con­trol­ler ob­jec­tives, take 2

  Maybe. Can you provide an argument for that?

  As stated, that wouldn’t maximize g, since applying the hash function once and tiling would cap the universe at finite depth. Tiling doesn’t make any sense.

Depth-based su­per­con­trol­ler ob­jec­tives, take 2

• sbenthallSep 23, 2014, 5:57 PM

  1 point

  in reply to: jimrandomh’s comment on: Depth-based su­per­con­trol­ler ob­jec­tives, take 2

  Your point about physical entropy is noted and a good one.

  One reason to think that something like D(u/​h) would pick out higher level features of reality is that h encodes those higher-level features. It may be possible to run a simulation of humanity on more efficient physical architecture. But unless that simulation is very close to what we’ve already got, it won’t be selected by g.

  You make an interesting point about the inefficiency of physics. I’m not sure what you mean by that exactly, and am not in a position of expertise to say otherwise. However, I think there is a way to get around this problem. Like Kolmogorov complexity, depth has another hidden term in it, the specification of the universal Turing machine that is used, concretely, to measure the depth and size of strings. By defining depth in terms of a universal machine that is a physics simulator, then there wouldn’t be a way to “bypass” physics computationally. That would entail being able to build a computer, which our physics, that would be more efficient than our physics. Tell me if that’s not impossible.

  Re: brains, I’m suggesting that we encode whatever we think is important about brains in the h term. If brains execute a computational process, then that process will be preserved somehow. It need not be preserve on grey matter exactly. Those brains could be uploaded onto more efficient architecture.

  I appreciate your intuitions on this but this function is designed rather specifically to challenge them.

• sbenthall 23 Sep 2014 17:47 UTC

  3 points

  in reply to: MrMind’s comment on: Depth-based su­per­con­trol­ler ob­jec­tives, take 2

  So, the key issue is whether or not the representations produced by the paperclip optimizer could have been produced by other processes. If there is another process that produces the paperclip-optimized representations more efficiently than going through the process of humanity, then that process dominates the calculation of D(r).

  In other words, for this objection to make sense, it’s not enough for the humanity to have been sufficient for the R scenario. It must be necessary for producing R, or at least necessary to result in it in the most efficient possible way.

  What are your criteria for a more concrete model than what has been provided?

• sbenthall 23 Sep 2014 0:07 UTC

  1 point

  on: CEV-tropes

  Do you know if there are literally entries for these outcomes on tvtropes.org? Should there be?

• sbenthall 23 Sep 2014 0:05 UTC

  1 point

  in reply to: CodeOtter’s comment on: Un­pop­u­lar ideas at­tract poor ad­vo­cates: Be charitable

  I think what the idea in the post does is that it gets at the curvature of the space, so to speak.

• sbenthall 22 Sep 2014 21:31 UTC

  0 points

  in reply to: slutbunwaller’s comment on: Every­body’s talk­ing about ma­chine ethics

  thanks I’ve been mispelling that for a while now. I stand corrected.

• sbenthall 22 Sep 2014 21:30 UTC

  1 point

  in reply to: Lumifer’s comment on: Every­body’s talk­ing about ma­chine ethics

  That is of course one of the questions on the table: who has the power to implement and promote different platforms.