rpglover64

• rpglover64Feb 17, 2023, 9:59 PM

  2 points

  0

  in reply to: Ben’s comment on: Are LLMs suffi­cient for AI take­off?

  But they seem like they are only doing part of the “intelligence thing”.

  I want to be careful here; there is some evidence to suggest that they are doing (or at least capable of doing) a huge portion of the “intelligence thing”, including planning, induction, and search, and even more if you include minor external capabilities like storage.

  I don’t know if anyone else has spoken about this, but since thinking about LLMs a little I am starting to feel like their something analagoss to a small LLM (SLM?) embedded somewhere as a component in humans

  I know that the phenomenon has been studied for reading and listening (I personally get a kick out of garden-path sentences); the relevant fields are “natural language processing” and “computational linguistics”. I don’t know know of any work specifically that addressed it in the “speaking” setting.

  if we want to build something “human level” then it stands to reason that it would end up with specialized components for the same sorts of things humans have specialized components for.

  Soft disagree. We’re actively building the specialized components because that’s what we want, not because that’s particularly useful for AGI.

[Question] Are LLMs suffi­cient for AI take­off?

• rpglover64Feb 17, 2023, 3:28 PM

  3 points

  2

  in reply to: Lost Futures’s comment on: Open & Wel­come Thread — Fe­bru­ary 2023

  Some thoughts:

  • Those who expect fast takeoffs would see the sub-human phase as a blip on the radar on the way to super-human

  • The model you describe is presumably a specialist model (if it were generalist and capable of super-human biology, it would plausibly count as super-human; if it were not capable of super-human biology, it would not be very useful for the purpose you describe). In this case, the source of the risk is better thought of as the actors operating the model and the weapons produced; the AI is just a tool

  • Super-human AI is a particularly salient risk because unlike others, there is reason to expect it to be unintentional; most people don’t want to destroy the world

  • The actions for how to reduce xrisk from sub-human AI and from super-human AI are likely to be very different, with the former being mostly focused on the uses of the AI and the latter being on solving relatively novel technical and social problems

• rpglover64Feb 14, 2023, 1:58 PM

  3 points

  3

  in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Cyborgism

  I think “sufficiently” is doing a lot of work here. For example, are we talking about >99% chance that it kills <1% of humanity, or >50% chance that it kills <50% of humanity?

  I also don’t think “something in the middle” is the right characterization; I think “something else” it more accurate. I think that the failure you’re pointing at will look less like a power struggle or akrasia and more like an emergent goal structure that wasn’t really present in either part.

  I also think that “cyborg alignment” is in many ways a much more tractable problem than “AI alignment” (and in some ways even less tractable, because of pesky human psychology):

  • It’s a much more gradual problem; a misaligned cyborg (with no agentic AI components) is not directly capable of FOOM (Amdhal’s law was mentioned elsewhere in the comments as a limit on usefulness of cyborgism, but it’s also a limit on damage)

  • It has been studied longer and has existed longer; all technologies have influenced human thought

  It also may be an important paradigm to study (even if we don’t actively create tools for it) because it’s already happening.

• rpglover64Feb 12, 2023, 7:41 PM

  1 point

  0

  AF

  in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Cyborgism

  Like, I may not want to become a cyborg if I stop being me, but that’s a separate concern from whether it’s bad for alignment (if the resulting cyborg is still aligned).

• rpglover64Feb 11, 2023, 9:40 PM

  1 point

  −6

  on: Cyborgism

  OpenAI’s focus with doing these kinds of augmentations is very much “fixing bugs” with how GPT behaves: Keep GPT on task, prevent GPT from making obvious mistakes, and stop GPT from producing controversial or objectionable content. Notice that these are all things that GPT is very poorly suited for, but humans find quite easy (when they want to). OpenAI is forced to do these things, because as a public facing company they have to avoid disastrous headlines like, for example: Racist AI writes manifesto denying holocaust.^[7]

  As alignment researchers, we don’t need to worry about any of that! The goal is to solve alignment, and as such we don’t have to be constrained like this in how we use language models. We don’t need to try to “align” language models by adding some RLHF, we need to use language models to enable us to actually solve alignment at its core, and as such we are free to explore a much wider space of possible strategies for using GPT to speed up our research.^[8]

  I think this section is wrong because it’s looking through the wrong frame. It’s true that cyborgs won’t have to care about PR disasters in their augments, but it feels like this section is missing the fact that there are actual issues behind the PR problem. Consider a model that will produce the statement “Men are doctors; women are nurses” (when not explicitly playing an appropriate role); this indicates something wrong with its epistemics, and I’d have some concern about using such a model.

  Put differently, a lot of the research on fixing algorithmic bias can be viewed as trying to repair known epistemic issues in the model (usually which are inherited from the training data); this seems actively desirable for the use case described here.

• rpglover64Feb 11, 2023, 3:08 PM

  LW: 7 AF: 3

  6

  AF

  in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Cyborgism

  I think that’s an important objection, but I see it applying almost entirely on a personal level. On the strategic level, I actually buy that this kind of augmentation (i.e. with in some sense passive AI) is not an alignment risk (any more than any technology is). My worry is the “dual use technology” section.

• rpglover64Feb 10, 2023, 5:36 PM

  3 points

  0

  in reply to: DragonGod’s comment on: On util­ity functions

  simple utility functions are fundamentally incapable of capturing the complexity/​subtleties/​nuance of human preferences.

  No objections there.

  that inexploitable agents exist that do not have preferences representable by simple/​unitary utility functions.

  Yep.

  Going further, I think utility functions are anti-natural to generally capable optimisers in the real world.

  I tentatively agree.

  That said

  The existence of a utility function is a sometimes useful simplifying assumption, in a way similar to how logical omniscience is (or should we be doing all math with logical inductors?), and naturally generalizes as a formalism to something like “the set of utility functions consistent with revealed preferences”.

  In the context of human rationality, I have found a local utility function perspective to be sometimes useful, especially as a probe into personal reasons; that is, if you say “this is my utility function” and then you notice “huh… my action does not reflect that”, this can prompt useful contemplation, some possible outcomes of which are:

  • You neglected a relevant term in the utility function, e.g. happiness of your immediate family

  • You neglected a relevant utility cost of the action you didn’t take, e.g. the aversiveness of being sweaty

  • You neglected a constraint, e.g. you cannot actually productively work for 12 hours a day, 6 days a week

  • The circumstances in which you acted are outside the region of validity of your approximation of the utility function, e.g. you don’t actually know how valuable having $100B would be to you

  • You made a mistake with your action

  Of course, a utility function framing is neither necessary nor sufficient for this kind of reflection, but for me, and I suspect some others, it is helpful.

• rpglover64Feb 10, 2023, 5:15 PM

  1 point

  0

  in reply to: Jozdien’s comment on: What’s ac­tu­ally go­ing on in the “mind” of the model when we fine-tune GPT-3 to In­struc­tGPT?

  The base models of GPT-3 already have the ability to “follow instructions”, it’s just veiled behind the more general interface. [...] you can see how it contains this capability somewhere.

  This is a good point that I forgot. My mental model of this is that since many training samples are Q&A, in these cases, learning to complete implies learning how to answer.

  InstructGPT also has the value of not needing the wrapper of “Q: [] A: []”, but that’s not really a qualitative difference.

  I want to push back a little bit on the claim that this is not a qualitative difference; it does imply a big difference in output for identical input, even if the transformation required to get similar output between the two models is simple.

  In other words, instruction following is not a new capability and the fine-tuning doesn’t really make any qualitative changes to the model. In fact, I think that you can get results [close to] this good if you prompt it really well (like, in the realm of soft prompts).

  TIL about soft prompts. That’s really cool, and I’m not surprised it works (it also feels a little related to my second proposed experiment). My intuition here (transferred from RNNs, but I think it should mostly apply to unidirectional transformers as well) is that a successful prompt puts the NN into the right “mental state” to generate the desired output: fine-tuning for e.g. instruction following mostly pushes to get the model into this state from the prompts given (as opposed to e.g. for HHH behavior, which also adjusts the outputs from induced states); soft prompts instead search for and learn a “cheat code” that puts the model into a state such that the prompt is interpreted correctly. Would you (broadly) agree with this?

• rpglover64Feb 10, 2023, 1:57 PM

  1 point

  0

  on: On util­ity functions

  Other people have given good answers to the main question, but I want to add just a little more context about self-modifying code.

  A bunch of MIRI’s early work explored the difficulties of the interaction of “rationality” (including utility functions induced by consistent preferences) with “self-modification” or “self-improvement”; a good example is this paper. They pointed out some major challenges that come up when an agent tries to reason about what future versions of itself will do; this is particularly important because one failure mode of AI alignment is to build an aligned AI that accidentally self-modifies into an unaligned AI (note that continuous learning is a restricted form of self-modification and suffers related problems). There are reasons to expect that powerful AI agents will be self-modifying (ideally self-improving), so this is an important question to have an answer to (relevant keywords include “stable values” and “value drift”).

  There’s also some thinking about self-modification in the human-rationality sphere; two things that come to mind are here and here. This is relevant because ways in which humans deviate from having (approximate, implicit) utility functions may be irrational, though the other responses point out limitations of this perspective.

• rpglover64Feb 10, 2023, 1:08 PM

  3 points

  0

  in reply to: DragonGod’s comment on: Open & Wel­come Thread—Jan­uary 2023

  Done: https://​​www.lesswrong.com/​​posts/​​eywpzHRgXTCCAi8yt/​​what-s-actually-going-on-in-the-mind-of-the-model-when-we

[Question] What’s ac­tu­ally go­ing on in the “mind” of the model when we fine-tune GPT-3 to In­struc­tGPT?

• rpglover64Feb 10, 2023, 12:38 AM

  3 points

  2

  on: Open & Wel­come Thread—Jan­uary 2023

  (I may promote this to a full question)

  Do we actually know what’s happening when you take an LLM trained on token prediction and fine-tune is via e.g. RLHF to get something like InstructGPT or ChatGPT? The more I think about the phenomenon, the more confused I feel.

  What links here?

  • What’s ac­tu­ally go­ing on in the “mind” of the model when we fine-tune GPT-3 to In­struc­tGPT? by rpglover64 (Feb 10, 2023, 7:57 AM; 18 points)

• rpglover64Feb 2, 2023, 11:24 PM

  8 points

  6

  on: Fo­cus on the places where you feel shocked ev­ery­one’s drop­ping the ball

  English doesn’t have great words for me to describe what I mean here, but it’s something like: your visualization machinery says that it sees no obstacle to success, such that you anticipate either success or getting a very concrete lesson.

  One piece of advice/​strategy I’ve received that’s in this vein is “maximize return on failure”. So prefer to fail in ways that you learn a lot, and to fail quickly, cheaply, and conclusively, and produce positive externalities from failure. This is not so much a good search strategy but a good guiding principle and selection heuristic.

• rpglover64Jan 31, 2023, 1:24 AM

  LW: 13 AF: 9

  3

  AF

  on: Align­ment al­lows “non­ro­bust” de­ci­sion-in­fluences and doesn’t re­quire ro­bust grading

  This is a great article! It helps me understand shard theory better and value it more; in particular, it relates to something I’ve been thinking about where people seem to conflate utility-optimizing agents with policy-execuing agents, but the two have meaningfully different alignment characteristics, and shard theory seems to be deeply exploring the latter, which is 👍.

  That is to say, prior to “simulators” and “shard theory”, a lot of focus was on utility-maximizers—agents that do things like planning or search to maximize a utility function; but planning, although instrumentally useful, is not strictly necessary for many intelligent behaviors, so we are seeing more focus on e.g. agents that enact learned policies in RL that do not explicitly maximize reward in deployment but try to enact policies that did so in training.

  The answer is not to find a clever way to get a robust grader. The answer is to not need a robust grader

  💯

  From my perspective, this post convincingly argues that one route to alignment involves splitting the problem into two still-difficult sub-problems (but actually easier, unlike inner- and outer-alignment, as you’ve said elsewhere): identifying a good shard structure and training an AI with such a shard structure. One point is that the structure is inherently somewhat robust (and that therefore each individual shard need not be), making it a much larger target.

  I have two objections:

  • I don’t buy the implied “naturally-robust” claim. You’ve solved the optimizer’s curse, wireheading via self-generated adversarial inputs, etc., but the policy induced by the shard structure is still sensitive to the details; unless you’re hiding specific robust structures in your back pocket, I have no way of knowing that increasing the candy-shard’s value won’t cause a phase shift that substantially increases the perceived value of the “kill all humans, take their candy” action plan. I ultimately care about the agent’s “revealed preferences”, and I am not convinced that those are smooth relative to changes in the shards.

  • I don’t think that we can train a “value humans” shard that avoids problems with the edge cases of what that means. Maybe it learns that it should kill all humans and preserve their history; or maybe it learns that it should keep them alive and comatose; or maybe it has strong opinions one way or another on whether uploading is death; or maybe it respects autonomy too much to do anything (though that one would probably be decomissioned and replaced by one more dangerous). The problem is not adversarial inputs but genuine vagueness where precision matters. I think this boils down to me disagreeing with John Wentworth’s “natural abstraction hypothesis” (at least in some ways that matter)

• rpglover64Jan 23, 2023, 3:31 PM

  1 point

  0

  on: De­con­fus­ing “Ca­pa­bil­ities vs. Align­ment”

  My working model for unintended capabilities improvement focuses more on second- and third-order effects: people see promise of more capabilities and invest more money (e.g. found startups, launch AI-based products), which increases the need for competitive advantage, which pushes people to search for more capabilities (loosely defined). There is also the direct improvement to the inner research loop, but this is less immediate for most work.

  Under my model, basically any technical work that someone could look at and think “Yes, this will help me (or my proxies) build more powerful AI in the next 10 years” is a risk, and it comes down to risk mitigation (because being paralyzed by fear of making the wrong move is a losing strategy):

  • One way to be safe is to take a very comms and policy flavored, helping highlight already known problems with AI, like Adversarial Policies Beat Superhuman Go AIs and Discovering Language Model Behaviors with Model-Written Evaluations

  • Another way is if the work just doesn’t apply to existing models, i.e. most MIRI work

• rpglover64Jan 20, 2023, 3:20 AM

  3 points

  0

  in reply to: johnswentworth’s comment on: The Plan − 2022 Update

  I don’t think “definitions” are the crux of my discomfort. Suppose the model learns a cluster; the position, scale, and shape parameters of this cluster summary are not perfectly stable—that is, they vary somewhat with different training data. This is not a problem on its own, because it’s still basically the same; however, the (fuzzy) boundary of the cluster is large (I have a vague intuition that the curse of dimensionality is relevant here, but nothing solid). This means that there are many cutting planes, induced by actions to be taken downstream of the model, on which training on different data could have yielded a different result. My intuition is that most of the risk of misalignment arises at those boundaries:

  • One reason for my intuition is that in communication between humans, difficulties arise in a similar way (i.e. when two peoples clusters have slightly different shapes)

  • One reason is that the boundary cases feel like the kind of stuff you can’t reliably learn from data or effectively test.

  Your comment seems to be suggesting that you think the edge cases won’t matter, but I’m not really understanding why the fuzzy nature of concepts makes that true.

• rpglover64Jan 11, 2023, 2:31 PM

  1 point

  0

  on: Com­pound­ing Re­source X

  This seems like a useful special case of “conditions-consequences” reasoning. I wonder whether

  • Avoiding meddling is a useful subskill in this context (probably not)

  • There is another useful special case

• rpglover64Jan 7, 2023, 12:44 AM

  1 point

  0

  in reply to: Noosphere89’s comment on: Con­tra Com­mon Knowledge

  A good example of this is the Collatz conjecture. It has a stupendous amount of evidence in the finite data points regime, but no mathematician worth their salt would declare Collatz solved, since it needs to have an actual proof.

  It’s important to distinguish the probability you’d get from a naive induction argument and a more credible one that takes into account the reference class of similar mathematical statements that hold until large but finite limits but may or may not hold for all naturals.

  Similarly, P=NP is another problem with vast evidence, but no proof.

  Arguably even more than the Collatz conjecture, if you believe Scott Aaronson (link).

  Mathematics is what happens when 0 or 1 are the only sets of probabilities allowed, that is a much stricter standard exists in math.

  As I mentioned in my other reply, there are domains of math that do accept probabilities < 1 as proof (but it’s in a very special way). Also, proof usually comes with insight (this is one of the reasons that the proof of the 4 color theorem was controversial), which is almost more valuable than the certainty it brings (and of course as skeptics and bayesians, we must refrain from treating it as completely certain anyway; there could have been a logical error that everyone missed).

  I think talking about proofs in terms of probabilities is a category error; logical proof is analogous to computable functions (equivalent if you are a constructivist), while probabilities are about bets.

• rpglover64Jan 7, 2023, 12:16 AM

  1 point

  0

  in reply to: abramdemski’s comment on: Con­tra Com­mon Knowledge

  I feel like this is a difference between “almost surely” and “surely”, both of which are typically expressed as “probability 1”, but which are qualitatively different. I’m wondering whether infinitesimals would actually work to represent “almost surely” as $1-ϵ$ (as suggested in this post).

  Also a nitpick and a bit of a tangent, but in some cases, a mathematician will accept any probability > 1 as proof; probabilistic proof is a common tool for non-constructive proof of existence, especially in combinatorics (although the way I’ve seen it, it’s usually more of a counting argument than something that relies essentially on probability).

  What links here?

  • rpglover64's comment on Con­tra Com­mon Knowledge by abramdemski (Jan 7, 2023, 12:44 AM; 1 point)