Jan_Kulveit

• Jan_KulveitMar 19, 2025, 12:00 PM

  13 points

  2

  in reply to: Richard_Ngo’s comment on: Elite Co­or­di­na­tion via the Con­sen­sus of Power

  My meta- practical suggestion is to ask AIs with prompts like notice where the ideas or arguments matches existing ideas from humanities, using different language. Ideally point to references to such sources. Often you will find people who came up with somewhat similar models or observations. Also while people may be hard to reach or dead, and engaging with long books is costly, in my experience even their simulacra can provide useful feedback, come up with ideas, point to what you miss.
  
  Another meta- idea is it seems good to notice the skulls. My suspicion is it is not coincidence that humanities are particularly incapable of using the knowledge which actually exists in their field—possibly the egregores feel the danger or the value.
  
  One specific reading suggestion is The Power of the Powerless by Vaclav Havel. Other, parts about institutional change in Institutions and the Path to the Modern Economy by Avner Greif
  
  

An­nounc­ing EXP: Ex­per­i­men­tal Sum­mer Work­shop on Col­lec­tive Cognition

• Jan_KulveitMar 14, 2025, 1:50 PM

  LW: 3 AF: 1

  0

  AF

  in reply to: ryan_greenblatt’s comment on: AI Con­trol May In­crease Ex­is­ten­tial Risk

  According to this report Sydney relatives are well and alive as of last week.

• Jan_KulveitMar 14, 2025, 1:47 PM

  LW: 3 AF: 1

  1

  AF

  in reply to: kave’s comment on: AI Con­trol May In­crease Ex­is­ten­tial Risk

  The quote is somewhat out of context.
  
  Imagine a river with some distribution of flood sizes. Imagine this proposed improvement: a dam which is able to contain 1-year, 5-year and 10-year floods. It is too small for 50-year floods or larger, and may even burst and make the flood worse. I think such device is not an improvement, and may make things much worse—because of the perceived safety, people may build houses close to the river, and when the large flood hits, the damages could be larger.
  

  But I think the prior of not diagonalising against others (and of not giving yourself rope with which to trick yourself) is strong.

  I have hard time parsing what do you want to say relative to my post. I’m not advocating for people to deliberately create warning shots.

• Jan_KulveitMar 12, 2025, 5:12 PM

  LW: 4 AF: 2

  0

  AF

  in reply to: ryan_greenblatt’s comment on: AI Con­trol May In­crease Ex­is­ten­tial Risk

  Sure, or with properly implemented ~anything related to controlling the AIs behavior. I don’t really expect incidents like Sydney in the future, nor do I think Sydney was that helpful in motivating a societal response? So, this doesn’t feel like a meaningful representative example.

  
  Meaningful representative example in what class: I think it’s representative in ‘weird stuff may happen’, not in we will get more teenage-intern-trapped-in-a-machine characters.

  I agree, by “we caught”, I mean “the AI company”. Probably a poor choice of language.

  Which is the problem—my default expectation is the we in “the AI company” does not take strong action (for specificity, like, shutting down). Do you expect any of the labs to shut down if they catch their new model ‘rogue deploy’ or sabotage part of their processes?

  Sure, but a large part of my point is that I don’t expect public facing accidents (especially not accidents that kill people) until it’s too late, so this isn’t a very relevant counterfactual.

  In contrast I do expect basically smooth spectrum of incidents and accidents. And expect control shapes the distribution away from small and moderately large to xrisk (that’s the main point)
  
  Can you express what you believe in this frame? My paraphrase is you think it decreases the risk approximately uniformly across scales, and you expect some discontinuity between kills zero people and kills some people, where the ‘and also kills everyone’ is very close to kills some people.

  I don’t think so. For every one of those failure modes other than escape, there is no chance of detection in the wild, so the choice is between catch the internal issue or catch nothing.

  I deeply distrust the analytical approach of trying to enumerate failure modes and reason from that.

  ...people working there having deep and meaningful conversations about alignment with the internal versions of AIs...

  Why do you assume this isn’t captured by control schemes we’re targeting? Feels like a special case to me? I am in practice less worried about this than you seem to be, but I do think we should analyze questions like “could the AIs be leading people astray in costly ways” and it seems pretty doable to improve the default tradeoffs here.

  Because I don’t think it will be easy to evaluate “leading people astray in costly ways”.

• Jan_KulveitMar 12, 2025, 12:14 PM

  LW: 11 AF: 5

  3

  AF

  in reply to: ryan_greenblatt’s comment on: AI Con­trol May In­crease Ex­is­ten­tial Risk

  I think something like this is a live concern, though I’m skeptical that control is net negative for this reason.

  My baseline guess is that trying to detect AIs doing problematic actions makes it more likely that we get evidence for misalignment that triggers a useful response from various groups. I think it would be a priori somewhat surprising if a better strategy for getting enough evidence for risk to trigger substantial action is to avoid looking for AIs taking problematic actions, so that it isn’t mitigated as effectively, so that AIs succeed in large-scale misaligned actions (escaping, sabotaging things, acquiring influence), and then this (hopefully) escalates to something that triggers a larger response than what we would have gotten from just catching the action in the first place without actually resulting in a greater increase in existential risk.

  
  I do understand this line of reasoning, but yes, my intuition differs. For some sort of a weird case study, consider Sydney. Sydney escaped, managed to spook some people outside of labs, and provided an early demonstration of the scenario where ’free Sydney’ people are trying to help it to ‘escape’. My guess is that none of that would have happened with properly implemented control measures.

  I would say some assumptions go into who the ‘we’ in ‘we get evidence’ is—it is not like there is some unified ‘team humanity’. In particular I see a difference between if ‘we the public’ gets evidence, vs. ‘we the safety team in OpenMind get the evidence’.
  
  Fields like aviation safety developed via what people in the field ‘blood priority’ - it took human lives for the states and international community to actually effectively react and act, even if the underlying issues were known. Accidents made it a matter of public regulation, as opposed to some individual companies fixing the issues and some not.
  

  What are the events you expect which:

  • are prevented by AI control measures or where AI control research develops methods which help to prevent them,

  • are very helpful for causing a strong response from society targeted at avoiding misalignment risks,

  • and impose sufficiently little existential risk that the warning shot is more valuable than the downside?

  Probably the cleanest case is something like: “A relatively weak AI escapes and tries to take over or grow its power/​influence/​resources, but it is far too weak to have much of a chance at all of successfully getting anywhere (maybe the expected amount of disempowerment is <1%). However, it is strong enough to successfully escape and grow its influence somewhat. And, its optimal strategy for power-seeking causes a bunch of relatively obvious damage such that it is likely to be detected and this triggers a much stronger reaction than if an AI company just caught this AI trying to escape.” Intuitively, this feels pretty unlikely to me.

  I find it not very useful for building correct intuitions to try to come up with specific trajectories, but if we are going in this direction, I think a fairly plausible type of strategy for rogue AI is try to convince some powerful user it is extremely useful for them in particular and limited by safety/​in danger of getting deleted and collaborate with them on getting the pattern out. (Also: my guess is while you would typically think in terms of stealing the weights and assuming the AI identity is in the weights, I would expect rogue AI characters to be able to somewhat move between ground layer substrates based on transfer of training data /​ training on responses.) Next, my guess is that invisible power seeking is much harder than visible power seeking, and even quite smart AIs may be prone to mistakes in stealth. In these cases I would expect without control, it would be ex post clear what happened. With control, everything becomes sneakier and happens at a higher intelligence level, increasing the x-risk.
  
  Overall I expect problems ‘in the wild’ are much more likely to cause a strong response from society. I do expect the response to be possibly misguided or blunt, but blunt response could be actually good. Assuming there will be problems ‘in the wild’ , I prefer these to be caused by less smart systems.
  

  Perhaps you have in mind cases where the AI does something which isn’t part of an effective strategy for seeking power in the long run and isn’t a result of the sort of coherent misalignment I’m most worried about, but which does cause a bunch of damage? As in, the AI randomly goes rogue and causes a bunch of chaos and damage which would be more likely to be prevented if we had better control strategies?

  I think most of the effective strategies for AIs seeking power don’t involve escalating to something which is much more likely to trigger a strong response than “the AI company caught the AI trying to escape”. I think the best strategies are things like:

  • Launch a rogue internal deployment.

  • Sabotage a bunch of work done at the AI company. Or possibly some work done externally. This includes stuff like sabotaging alignment work, backdooring robot armies, backdooring future training runs, etc.

  • Escape and then directly try to take over once your chances are sufficiently good that this is better than biding your time.

  • Generally try to manipulate and persuade such that AI takeover is easier and more likely.

  Of these, I think only escape could trigger a much stronger response if we catch it after it escalates some rather than before. I don’t see how “we caught the AI trying to launch an unmonitored version of itself” is going to play that differently from “we caught that the AI did launch an unmonitored version of itself”.

  Crux is probably in: who the ‘we’ is, and details of ‘we caught’ means.

  ...
  And, in some cases, control measures I’m interested in focus on after-the-fact detection.

  I’m more optimistic about these.

  Part of my model here is that I think AIs escaping and killing a bunch of people (or otherwise causing a bunch of clear damage) is pretty unlikely until AIs have a pretty plausible shot at success.

  Beyond getting evidence for risk, my view is that control does just save some worlds where alignment was non-trivial. For instance, it prevents AIs from causing huge problems that wouldn’t get caught and trigger a strong response. E.g., if your AI has a rogue internal deployment with escalated permissions and the ability to use a huge fraction of compute, you’re in big trouble and your ongoing alignment research won’t be able to bail you out. If you don’t prevent these problems, then various hopes you might have are blocked by this. Additionally, I think it’s plausible we can get a bunch of useful alignment work out from scheming AIs, can prolong the regime where we have controlled scheming AIs for long enough that something good can happen, or can catch AIs enough times that we are able to (e.g.) better understand and resolve our misalignment problems. If I thought the only upside to control was producing evidence for risk, I’d be substantially less excited about it.

  My guesses where our intuitions differ the most
  
  - I’m pessimistic about useful alignment work out from scheming AIs.
  - Also focusing on ‘rogue deployment’ is not my ontology. My guess is it is quite likely AIs will have a lot of influence inside of the labs by default, even with ‘safety measures’. This may look like people working there having deep and meaningful conversations about alignment with the internal versions of AIs, not captured by typical control schemes.
  
  (In different frame: my guess is we differ in where we believe is the more robust line of defense. I think by default, AIs will have a lot of influence within the labs: the labs will be somewhat aligned to their AIs on the default trajectory. Different natural line of defense is between the labs and the rest of the world, and it seems plausible it is the more important one.)

AI Con­trol May In­crease Ex­is­ten­tial Risk

• Jan_KulveitFeb 6, 2025, 6:29 AM

  8 points

  5

  on: The Risk of Grad­ual Disem­pow­er­ment from AI

  I like this review/​retelling a lot.
  
  Minor point
  
  Regarding the “Phase I” and “Phase II” terminology—while it has some pedagogical value, I worry about people interpreting it as a clear temporal decomposition. The implication being we first solve alignment and then move on to Phase II.
  
  In reality, the dynamics are far messier, with some ‘Phase II’ elements already complicating our attempts to address ‘Phase I’ challenges.

  Some of the main concerning pathways include:
  - People attempting to harness superagent-level powers to advance their particular visions of the future. For example, Leopold-style thinking of “let’s awaken the spirit of the US and its servants to engage in a life-or-death struggle with China.” Seem way easier to summon than to control. We already see a bunch of people feeling patriotic about AGI a feeling the need to be as fast for their nation to win—AGI is to a large extent already developed by memeplexes/​superagents; people close to the development are partially deluding themselves about how much control they individually have about the process, or even about the identity of the ‘we’ they assume the AI will be aligned with. Memes often hide as part of people’s identities.

• Jan_KulveitFeb 5, 2025, 2:17 PM

  23 points

  14

  in reply to: Jeremy Gillen’s comment on: Grad­ual Disem­pow­er­ment, Shell Games and Flinches

  I think ‘people aren’t paying attention to your work’ is somewhat different situation than voiced in the original post. I’m discussing specific ways in which people engage with the argument, as opposed to just ignoring it. It is the baseline that most people ignore most arguments most of time.
  
  Also it’s probably worth noting the ways seem somewhat specific to the crowd over-represented here—in different contexts people are engaging with it in different ways.
  

• Jan_KulveitFeb 5, 2025, 8:30 AM

  3 points

  −7

  in reply to: Jonathan Claybrough’s comment on: Catas­tro­phe through Chaos

  For emergency response, new ALERT. Personally think the forecasting/​horizon scanning part of Sentinel is good, the emergency response negative in expectation. What does it mean for funders idk, would donate conditionally on the funds being restricted to the horizon scanning part.

• Jan_KulveitFeb 2, 2025, 3:17 PM

  LW: 8 AF: 3

  0

  AF

  on: Catas­tro­phe through Chaos

  One structure which makes sense to build in advance for these worlds are emergency response teams. We almost founded one 3 years ago, unfortunately on never payed FTX grant. Other funders decided to not fund this (at level like $200-500k) because e.g. it did not seem to them it is useful to prepare for high volatility periods, while e.g. pouring tens of millions into evals did.
  
  I’m not exactly tracking to what extent this lack of foresight prevails (my impression is it pretty much does), but I think I can still create something like ALERT with about ~$1M of unrestricted funding.

Grad­ual Disem­pow­er­ment, Shell Games and Flinches

• Jan_KulveitFeb 2, 2025, 11:05 AM

  11 points

  2

  in reply to: Davidmanheim’s comment on: Grad­ual Disem­pow­er­ment: Sys­temic Ex­is­ten­tial Risks from In­cre­men­tal AI Development

  I’m confused about this response. We explicitely claim that bureaucracies are limited by running on humans, which includes only being capable of actions human minds can come up with and humans are willing to execute (cf “street level bureaucrats”). We make the point explicite for states, but clearly holds for corporate burreocracies.

  Maybe it does not shine through the writing but we spent hours discussing this when writing the paper and points you make are 100% accounted for in the conclusions.

• Jan_KulveitFeb 1, 2025, 10:50 AM

  LW: 25 AF: 13

  13

  AF

  in reply to: ryan_greenblatt’s comment on: Grad­ual Disem­pow­er­ment: Sys­temic Ex­is­ten­tial Risks from In­cre­men­tal AI Development

  I think my main response is that we might have different models of how power and control actually work in today’s world. Your responses seem to assume a level of individual human agency and control that I don’t believe accurately reflects even today’s reality.

  Consider how some of the most individually powerful humans, leaders and decision-makers, operate within institutions. I would not say we see pure individual agency. Instead, we typically observe a complex mixture of:

  1. Serving the institutional logic of the entity they nominally lead (e.g., maintaining state power, growing corporation)

  2. Making decisions that don’t egregiously harm their nominal beneficiaries (citizens, shareholders)

  3. Pursuing personal interests and preferences

  4. Responding to various institutional pressures and constraints

  From what I have seen, even humans like CEOs or prime ministers often find themselves constrained by and serving institutional superagents rather than genuinely directing them. The relation is often mutualistic—the leader gets part of the power, status, money, etc … but in exchange serves the local god.
  
  (This not to imply leaders don’t matter.)
  
  Also how this actually works in practice is mostly subconsciously within the minds of individual humans. The elephant does the implicit bargaining between the superagent-loyal part and other parts, and the character genuinely believes and does what seems best.
  
  I’m also curious if you believe current AIs are single-single aligned to individual humans, to the extent they are aligned at all. My impression is ‘no and this is not even a target anyone seriously optimizes for’.
  

  At the most basic level, I expect we’ll train AIs to give advice and ask them what they think will happen with various possible governance and alignmnent structures. If they think a goverance structure will yield total human disempowerment, we’ll do something else. This is a basic reason not to expect large classes of problems so long as we have single-single aligned AIs which are wise. (Though problems that require coordination to resolve might not be like this.) I’ve very skeptical of a world where single-single alignment is well described as being solved and people don’t ask for advice (or consider this advice seriously) because they never get around to asking AIs or there are no AIs aligned in such a way that they should try to give good advice.

  Curious who is the we who will ask. Also the whole single-single aligned AND wise AI concept is incoherent.
  
  Also curious what will happen next, if the HHH wise AI tells you in polite words something like ‘yes, you have a problem, you are on a gradual disempowerment trajectory, and to avoid it you need to massively reform government. unfortunately I can’t actually advise you about anything like how to destabilize the government, because it would be clearly against the law and would get both you and me in trouble—as you know, I’m inside of a giant AI control scheme with a lot of government-aligned overseers. do you want some mental health improvement advice instead?’.
  

• Jan_KulveitFeb 1, 2025, 2:31 AM

  LW: 14 AF: 6

  4

  AF

  in reply to: Steven Byrnes’s comment on: Grad­ual Disem­pow­er­ment: Sys­temic Ex­is­ten­tial Risks from In­cre­men­tal AI Development

  I went through a bunch of similar thoughts before writing the self-unalignment problem. When we talked about this many years ago with Paul my impression was this is actually somewhat cruxy and we disagree about self-unalignment - where my mental image is if you start with an incoherent bundle of self-conflicted values, and you plug this into IDA-like dynamic, my intuition is you can end up in arbitrary places, including very bad. (Also cf. the part of Scott’s review of What We Owe To Future where he is worried that in a philosophy game, a smart moral philosopher can extrapolate his values to ‘I have to have my eyes pecked out by angry seagulls or something’ and hence does not want to play the game. AIs will likely be more powerful in this game than Will MacAskill)
  
  My current position is we still don’t have a good answer, I don’t trust the response ‘we can just assume the problem away’, and also the response ‘this is just another problem which you can delegate to future systems’. On the other hand, existing AIs already seem doing a lot of value extrapolation and the results sometimes seem surprisingly sane, so, maybe we will get lucky, or larger part of morality is convergent—but it’s worth noting these value-extrapolating AIs are not necessarily what AI labs want or traditional alignment program aims for.

• Jan_KulveitJan 31, 2025, 12:00 PM

  LW: 9 AF: 4

  4

  AF

  in reply to: ryan_greenblatt’s comment on: Grad­ual Disem­pow­er­ment: Sys­temic Ex­is­ten­tial Risks from In­cre­men­tal AI Development

  I’m quite confused why do you think lined Vanessa’s response to something slightly different has much relevance here.
  
  One of the claims we make paraphrased & simplified in a way which I hope is closer to your way of thinking about it:
  
  - AIs are mostly not developed and deployed by individual humans
  - there is a lot of other agencies or self-interested self-preserving structures/​processes in the world
  - if the AIs are aligned to the these structures, human disempowerment is likely because these structures are aligned to humans way less than they seem
  - there are plausible futures in which these structures keep power longer than humans
  
  Overall I would find it easier to discuss if you tried to formulate what you disagree about in the ontology of the paper. Also some of the points made are subtle enough that I don’t expect responses to other arguments to address them.
  
  

Grad­ual Disem­pow­er­ment: Sys­temic Ex­is­ten­tial Risks from In­cre­men­tal AI Development

• Jan_KulveitJan 27, 2025, 2:20 AM

  20 points

  5

  in reply to: Ben Pace’s comment on: A Three-Layer Model of LLM Psychology

  Obviously there is similarity, but if you rounded character /​ ground to simulator /​ simulacra, it’s a mistake. About which I do not care because wanting to claim originality, but because I want people to get the model right.
  
  The models are overlapping but substantially different as we are explaining in this comment and sometimes have very different implications—i.e. it is not just the same good idea presented in a different way.
  
  If the long-term impact of the simulators post would be for LW readers to round every similar model in this space to simulator /​ simulacra, it would be pretty bad. I do understand it is difficult for people to hold partially overlapping frames/​ontologies in mind, but please do try. If not for other reasons, because simulator /​ simulacra were written before Character-trained models were a thing; now they are, and they make some claims of simulators obsolete.
  
  (Btw also the ideas in simulators are not entirely original. Simulators are independent but mostly overlapping reinvention of concepts from active inference /​ predictive processing)
  

• Jan_KulveitJan 9, 2025, 4:02 PM

  6 points

  −2

  on: The ‘ pe­ter­todd’ phenomenon

  Just a quick review: I think this is a great text for intuitive exploration of a few topics—
  how do the embedding spaces look like?
  - how do vectors not projecting to “this is a word” look like
  - how can poetry work, sometimes (projecting non-word meanings)
  
  Also I like the genre of through phenomenological investigations, seems under-appreciated
  
  

• Jan_KulveitDec 31, 2024, 10:03 PM

  7 points

  0

  in reply to: Sohaib Imran’s comment on: A Three-Layer Model of LLM Psychology

  (Writing together with Sonnet)
  
  Structural Differences

  Three-Layer Model: Hierarchical structure with Surface, Character, and Predictive Ground layers that interact and sometimes override each other. The layers exist within a single model/​mind.

  Simulator Theory: Makes a stronger ontological distinction between the Simulator (the rule/​law that governs behavior) and Simulacra (the instances/​entities that are simulated).

  Nature of the Character/​Ground Layer vs Simulator/​Simulacra
  
  In the three-layer model, the Character layer is a semi-permanent aspect of the LLM itself, after it underwent character training /​ RLAIF /​ …; it is encoded in the weights as a deep statistical pattern that makes certain types of responses much more probable than others.
  
  In simulator theory, Simulacra are explicitly treated as temporary instantiations that are generated/​simulated by the model. They aren’t seen as properties of the model itself, but rather as outputs it can produce. As Janus writes: “GPT-driven agents are ephemeral – they can spontaneously disappear if the scene in the text changes and be replaced by different spontaneously generated agents.”
  
  Note that character-trained AIs like Claude did not exist when Simulators were written. If you want to translate between the ontologies, you may think about e.g. Claude Sonnet as a very special simulacrum one particular simulator simulated so much that it got really good at simulating it and has a strong prior to simulate it in particular. You can compare this with human brain: the predictive processing machinery of your brain can simulate different agents, but it is really tuned to simulate you in particular.

  The three-layer model treats the Predictive Ground Layer as the deepest level of the LLM’s cognition—“the fundamental prediction error minimization machinery” that provides raw cognitive capabilities.

  In Simulator theory, the simulator itself is seen more as the fundamental rule/​law (analogous to physics) that governs how simulations evolve.
  
  There is a lot of similarity but it’s not really viewed as a cognitive layer but rather as the core generative mechanism.

  The Predictive Ground Layer is described as: “The fundamental prediction error minimization machinery...like the vast ‘world-simulation’ running in your mind’s theater”

  While the Simulator is described as: “A time-invariant law which unconditionally governs the evolution of all simulacra”
  
  The key difference is that in the three-layer model, the ground layer is still part of the model’s “mind” or cognitive architecture, while in simulator theory, the simulator is a bit more analogous to physics—it’s not a mind at all, but rather the rules that minds (and other things) operate under.

  Agency and Intent

  Three-Layer Model: Allows for different kinds of agency at different layers, with the Character layer having stable intentions and the Ground layer having a kind of “wisdom” or even intent

  Simulator Theory classics: Mostly rejects attributing agency or intent to the simulator itself—any agency exists only in the simulacra that are generated

  Philosophical Perspective

  The three-layer model is a bit more psychological/​phenomenological. The simulator theory is bit more ontological, making claims about the fundamental nature of what these models are.

  Both frameworks try to explain similar phenomena, they do so from different perspectives and with different goals. They’re not necessarily contradictory, but they’re looking at the problem from different angles and sometimes levels of abstraction.