IAFF-User-111

• IAFF-User-111Jan 3, 2017, 3:19 AM

  0 points

  AF

  in reply to: jessicata’s comment on: My cur­rent take on the Paul-MIRI dis­agree­ment on al­ignabil­ity of messy AI

  I don’t see this as being the case. As Vadim pointed out, we don’t even know what we mean by “aligned versions” of algos, ATM. So we wouldn’t know if we’re succeeding or failing (until it’s too late and we have a treacherous turn).

  It looks to me like Wei Dai shares my views on “safety-performance trade-offs” (grep it here: http://​​graphitepublications.com/​​the-beginning-of-the-end-or-the-end-of-beginning-what-happens-when-ai-takes-over/​​).

  I’d paraphrase what he’s said as:

  “Orthogonality implies that alignment shouldn’t cost performance, but says nothing about the costs of ‘value loading’ (i.e. teaching an AI human values and verifying its value learning procedure and/​or the values it has learned). Furthermore, value loading will probably be costly, because we don’t know how to do it, competitive dynamics make the opportunity cost of working on it large, and we don’t even have clear criteria for success.”

  Which I emphatically agree with.

• IAFF-User-111Jan 3, 2017, 3:19 AM

  0 points

  AF

  in reply to: jessicata’s comment on: My cur­rent take on the Paul-MIRI dis­agree­ment on al­ignabil­ity of messy AI

  I don’t see this as being the case. As Vadim pointed out, we don’t even know what we mean by “aligned versions” of algos, ATM. So we wouldn’t know if we’re succeeding or failing (until it’s too late and we have a treacherous turn).

  It looks to me like Wei Dai shares my views on “safety-performance trade-offs” (grep it here: http://​​graphitepublications.com/​​the-beginning-of-the-end-or-the-end-of-beginning-what-happens-when-ai-takes-over/​​).

  I’d paraphrase what he’s said as:

  “Orthogonality implies that alignment shouldn’t cost performance, but says nothing about the costs of ‘value loading’ (i.e. teaching an AI human values and verifying its value learning procedure and/​or the values it has learned). Furthermore, value loading will probably be costly, because we don’t know how to do it, competitive dynamics make the opportunity cost of working on it large, and we don’t even have clear criteria for success.”

  Which I emphatically agree with.

• IAFF-User-111Aug 30, 2016, 7:34 PM

  LW: 1 AF: 1

  AF

  on: Sim­plified ex­pla­na­tion of stratification

  So after talking w/​Stuart, I guess what he means by “humans learning from the AI’s actions” is that what humans’ beliefs about U converges to actually changes (for the better). I’m not sure if that’s really desirable, atm.

  On a separate note, my proposal has the practical issue that this agent only views its own potential influence on u* as undesirable (and not other agents’). So I think ultimately we want a more rich set of counter-factuals, including, e.g. that humans continue to exist indefinitely (otherwise P_Ht becomes undefined when humanity is extinct).

• IAFF-User-111Aug 30, 2016, 6:17 PM

  LW: 3 AF: 2

  AF

  on: Sim­plified ex­pla­na­tion of stratification

  Abstractly, I think of this as adding a utility node, U, with no parents, and having the agent try to maximize the expected value of U.

  I think there are some implicit assumptions (which seem reasonable for many situations, prime facie) about the agent’s ability to learn about U via some observations when taking null actions (i.e. A and U share some descendant(s), D, and A knows something about P(D | U, A=null).

  RE: the last bit, it seems like you can define learning from manipulating in a straightforward way similar to what is proposed here. The intuition is that the humans belief about U should be collapsing around a point, u* (in the absence of interference by the AI), and the AI helps learning if it accelerates this process. If this is literally true, then we can just say that learning is accelerated (at tstep t) if the probability H assigns to u* is higher given an agents action a than it would be given the null action, i.e.

  P_H_t(u* | A_0 = a) > P_H_t(u* | A_0 = A1 = … = null).

• IAFF-User-111Aug 30, 2016, 12:10 AM

  0 points

  AF

  on: Learn­ing val­ues ver­sus indifference

  RE: my last question- After talking to Stuart, I think one way of viewing the problem with such a proposal is: The agent cares about its future expected utility (which depends on the state/​history, not just the MDP).

• IAFF-User-111Aug 26, 2016, 8:51 PM

  0 points

  AF

  on: Learn­ing val­ues ver­sus indifference

  Why doesn’t normalizing rewards work?

  (i.e. set max_pi(expected returns)=1 and min_pi(expected_returns)=0, for all environments)… I assume this is what you’re talking about at the end?

• IAFF-User-111Feb 11, 2016, 9:40 PM

  LW: 2 AF: 1

  AF

  in reply to: paulfchristiano’s comment on: Some work on con­nect­ing UDT and Re­in­force­ment Learning

  “But in the setting you described, the only impact of the policy is on the agent’s actions”

  I don’t think so. P_M(\zeta | \pi) is meant to describe the distribution over trajectories given a policy (according to the model). Unless I’m missing something, the model could contain non-causal correlations.

• IAFF-User-111Jan 29, 2016, 6:09 AM

  0 points

  AF

  on: Re­source gath­er­ing agent

  Doesn’t seem workable to me: being “completely ignorant” suggests an improper prior. An agent with a proper prior over its utility function can integrate over it and maximize expected utility and which action maximizes expected utility will depend on this prior.

• IAFF-User-111Jan 28, 2016, 12:31 AM

  0 points

  AF

  on: New(ish) AI con­trol ideas

  Thanks! I love having central repos.

  A quick question /​ comment, RE: “I decided to try and attack as many of these ideas as I could, head on, and see if there was any way of turning these objections.”

  Q: What do you mean (or have in mind) in terms of “turning [...] objections”? I’m not very familiar with the phrase.

  Comment: One trend I see is that technical safety proposals are often dismissed by appealing to one of the 7 responses you’ve given. Recently I’ve been thinking that we should be a bit less focused on finding airtight solutions, and more focused on thinking about which proposed techniques could be applied in various scenarios to significantly reduce risk. For example, boxing an agent (e.g. by limiting it’s sensors/​actuators) might significantly increase how long it takes to escape.

• IAFF-User-111Jan 28, 2016, 12:06 AM

  0 points

  AF

  on: At­tempt­ing to re­fine “max­i­miza­tion” with 3 new -izers

  skimmed it.

  It would be helpful to define “stopping point” and “stopping distance”.

  ---

  Wrt local optima:

  Deep Neural Nets were historically thought to suffer from local optima. Recently, this viewpoint has been challenged; see, e.g. “The Loss Surfaces of Multilayer Networks” http://​​arxiv.org/​​abs/​​1412.0233 and references.

  Although the issue remains unclear, I currently suspect that local optima are not a practical obstacle for an (omniscient) hill-climber in the real world.

  ---

  I wasn’t convinced overall by the statement about tiling (or not). I think you should give more detailed arguments about why you do or don’t expect these agents to tile, and explain the set-up a bit more, too: are you imagining agents that take a single action, based on their current policy, to adopt a new policy, which is then not subject to further modification? Or how can you ensure that agents do not modify their policy in such a way that policy_new encourages further modifications which can compound?

• IAFF-User-111Jan 27, 2016, 11:53 PM

  0 points

  AF

  in reply to: paulfchristiano’s comment on: Some work on con­nect­ing UDT and Re­in­force­ment Learning

  I don’t understand why you say:

  1. it “seems to require a richer model than we usually use in [RL]”.

  2. “This seems to happen in your setting.”

  3. Are you suggesting that a model as I’ve defined it is not satisfactory/​sufficient for some reason?

  4. can you elaborate a bit?

Some work on con­nect­ing UDT and Re­in­force­ment Learning

UDT from an RL perspective

• IAFF-User-111Oct 2, 2015, 3:01 AM

  LW: 2 AF: 1

  AF

  in reply to: jessicata’s comment on: Pro­posal: Model­ing goal sta­bil­ity in ma­chine learning

  So the big problem I see with this it is still in the optimization framework, assuming that we actually want to optimize the initial criterion. While we can imagine changing the initial criterion, this is already something we can effectively do with RL if we specify our reward to be something communicated by a human overseer (but of course that doesn’t really solve the problem...)

  The proposal is reminiscent of the Actor-Critic framework from RL (analogy: actor—model, critic—criterion), which learns a policy (the actor) and a value function (the critic) simultaneously.

  In that case, you have the true reward function playing the role of the initial criterion, so you don’t actually get to evaluate the true criterion (which would be something like distance from the optimal policy), you get what amounts to noisy samples of it. The goal in both cases is to learn a good model (i.e. policy, for Actor-Critic).

  I think there is a conceptual issue with this proposal as it stands, namely, the interplay between the changes in the model and criterion are not taken into account. E.g. there is no guarantee that recursively applying F to the initial_model using the criteria output by X would give you anything like the model output by X.

  The cool thing about Actor-Critic is that you can prove (under suitable assumptions) that this method actually gives you an unbiased estimate of the true policy gradient (Sutton 99: https://​​webdocs.cs.ualberta.ca/​​~sutton/​​papers/​​SMSM-NIPS99.pdf). IIRC, it requires the assumption that the critic is trained to convergence in-between each update of the actor, though.