Jan_Kulveit

• Jan_Kulveit 14 Nov 2024 0:00 UTC

  14 points

  0

  on: Jan_Kul­veit’s Shortform

  Central european experience, which is unfortunately becoming relevant also for the current US: for world-modelling purposes, you should have hypotheses like ‘this thing is happening because of a russian intelligence operation’ or ‘this person is saying what they are saying because they are a russian asset’ in your prior with nontrivial weights.

Jan_Kul­veit’s Shortform

• Jan_Kulveit 16 Oct 2024 19:58 UTC

  14 points

  0

  on: Against em­pa­thy-by-default

  I expected quite different argument for empathy
  
  1. argument from simulation: most important part of our environment are other people; people are very complex and hard to predict; fortunately, we have a hardware which is extremely good at ‘simulating a human’ - our individual brains. to guess what other person will do or why they are doing what they are doing, it seems clearly computationally efficient to just simulate their cognition on my brain. fortunately for empathy, simulations activate some of the same proprioceptive machinery and goal-modeling subagents, so the simulation leads to similar feelings
  
  2. mirror neurons: it seems we have powerful dedicated system for imitation learning, which is extremely advantageous for overcoming genetic bottleneck. mirroring activation patterns leads to empathy

• Jan_Kulveit 2 Oct 2024 7:46 UTC

  11 points

  7

  in reply to: Cleo Nardo’s comment on: straw­berry calm’s Shortform

  My personal impression is you are mistaken and the innovation have not stopped, but part of the conversation moved elsewhere. E.g. taking just ACS, we do have ideas from past 12 months which in our ideal world would fit into this type of glossary—free energy equilibria, levels of sharpness, convergent abstractions, gradual disempowerment risks. Personally I don’t feel it is high priority to write them for LW, because they don’t fit into the current zeitgeist of the site, which seems directing a lot of attention mostly to:
  - advocacy—
  topics a large crowd cares about (e.g. mech interpretability)
  - or topics some prolific and good writer cares about (e.g. people will read posts by John Wentworth)
  Hot take, but the community loosely associated with active inference is currently better place to think about agent foundations; workshops on topics like ‘pluralistic alignment’ or ‘collective intelligence’ have in total more interesting new ideas about what was traditionally understood as alignment; parts of AI safety went totally ML-mainstream, with the fastest conversation happening at x.
  

• Jan_Kulveit 10 Sep 2024 6:33 UTC

  LW: 33 AF: 20

  21

  AF

  on: AI fore­cast­ing bots incoming

  Seems worth mentioning SOTA, which is https://​​futuresearch.ai/​​. Based on the competence & epistemics of Futuresearch team and their bot get very strong but not superhuman performance, roll to disbelieve this demo is actually way better and predicts future events at superhuman level.
  
  Also I think it is a generally bad to not mention or compare to SOTA but just cite your own prior work. Shame.

• Jan_Kulveit 25 Jul 2024 12:37 UTC

  16 points

  −2

  in reply to: Arthur Conmy’s comment on: You should go to ML conferences

  I’m skeptical of the ‘wasting my time’ argument.

  Stance like ‘going to poster sessions is great for young researchers, I don’t do it anymore and just meet friends’ is high-status, so, on priors, I would expect people to take it more than what’s optimal.

  Realistically, poster session is ~1.5h, maybe 2h with skimming what to look at. It is relatively common for people in AI to spend many hours per week digesting what are the news on twitter. I really doubt the per hour efficiency of following twitter is better than of poster sessions when approached intentionally. (While obviously aimlessly wandering between endless rows of posters is approximately useless.)

• Jan_Kulveit 24 Jul 2024 16:43 UTC

  2 points

  0

  in reply to: niplav’s comment on: You should go to ML conferences

  Corrected!

• Jan_Kulveit 24 Jul 2024 12:00 UTC

  9 points

  0

  on: The last era of hu­man mistakes

  I broadly agree with this—we tried to describe somewhat similar set of predictions in Cyborg periods.

You should go to ML conferences

• Jan_Kulveit 2 Jul 2024 21:15 UTC

  5 points

  0

  on: List of Col­lec­tive In­tel­li­gence Projects

  Surprised you haven’t heard about any facilitated communication tools.

• Jan_Kulveit 24 Jun 2024 22:23 UTC

  34 points

  6

  on: LLM Gen­er­al­ity is a Timeline Crux

  Few thoughts
  - actually, these considerations mostly increase uncertainty and variance about timelines; if LLMs miss some magic sauce, it is possible smaller systems with the magic sauce could be competitive, and we can get really powerful systems sooner than Leopold’s lines predict
  - my take on what is one important thing which makes current LLMs different from humans is the gap described in Why Simulator AIs want to be Active Inference AIs; while that post intentionally avoids having a detailed scenario part, I think the ontology introduced is better for thinking about this than scaffolding
  - not sure if this is clear to everyone, but I would expect the discussion of unhobbling being one of the places where Leopold would need to stay vague to not breach OpenAI confidentiality agreements; for example, if OpenAI was putting a lot of effort into make LLM-like systems be better at agency, I would expect he would not describe specific research and engineering bets

The Liv­ing Planet In­dex: A Case Study in Statis­ti­cal Pitfalls

• Jan_Kulveit 19 Jun 2024 13:20 UTC

  1 point

  0

  in reply to: TsviBT’s comment on: TsviBT’s Shortform

  Agreed we would have to talk more. I think I mostly get the homunculi objection. Don’t have time now to write an actual response, so here are some signposts:
  - part of what you call agency is explained by roughly active inference style of reasoning
  —some type of “living” system is characteristic by having boundaries between them and the environment (boundaries mostly in sense of separation of variables)
  -- maintaining the boundary leads to need to model the environment
  —modelling the environment introduces a selection pressure toward approximating Bayes
  - other critical ingredient is boundedness
  —in this universe, negentropy isn’t free
  —this introduces fundamental tradeoff /​ selection pressure for any cognitive system: length isn’t free, bitflips aren’t free, etc.
  (--- downstream of that is compression everywhere, abstractions)
  -- empirically, the cost/​returns function for scaling cognition usually hits diminishing returns, leading to minds where it’s not effective to grow the single mind further
  —this leads to the basin of convergent evolution I call “specialize and trade”—
  empirically, for many cognitive systems, there is a general selection pressure toward modularity
  —I don’t know what are all the reasons for that, but one relatively simple is ‘wires are not free’; if wires are not free, you get colocation of computations like brain regions or industry hubs
  —other possibilities are selection pressures from CAP theorem, MVG, …
  (modularity also looks a bit like box-inverted specialize and trade)
  
  So, in short, I think where I agree with the spirit of If humans didn’t have a fixed skull size, you wouldn’t get civilization with specialized members and my response is there seems to be extremely general selection pressure in this direction. If cells were able to just grow in size and it was efficient, you wouldn’t get multicellulars. If code bases were able to just grow in size and it was efficient, I wouldn’t get a myriad of packages on my laptop, it would all be just kernel. (But even if it was just kernel, it seems modularity would kick in and you still get the ‘distinguishable parts’ structure.)

• Jan_Kulveit 17 Jun 2024 1:23 UTC

  0 points

  0

  in reply to: TsviBT’s comment on: TsviBT’s Shortform

  That’s why solving hierarchical agency is likely necessary for success

• Jan_Kulveit 5 Jun 2024 9:17 UTC

  91 points

  69

  on: Former OpenAI Su­per­al­ign­ment Re­searcher: Su­per­in­tel­li­gence by 2030

  (crossposted from twitter) Main thoughts:
  1. Maps pull the territory
  2. Beware what maps you summon
  
  Leopold Aschenbrenners series of essays is a fascinating read: there is a ton of locally valid observations and arguments. Lot of the content is the type of stuff mostly discussed in private. Many of the high-level observations are correct.

  At the same time, my overall impression is the set of maps sketched pulls toward existential catastrophe, and this is true not only for the ‘this is how things can go wrong’ part, but also for the ‘this is how we solve things’ part. Leopold is likely aware of the this angle of criticism, and deflects it with ‘this is just realism’ and ‘I don’t wish things were like this, but they most likely are’. I basically don’t buy that claim.

An­nounc­ing Hu­man-al­igned AI Sum­mer School

• Jan_Kulveit 17 May 2024 11:31 UTC

  3 points

  0

  on: The Align­ment Prob­lem No One Is Talk­ing About

  You may be interested in ‘The self-unalignment problem’ for some theorizing https://​​www.lesswrong.com/​​posts/​​9GyniEBaN3YYTqZXn/​​the-self-unalignment-problem

• Jan_Kulveit 24 Apr 2024 14:04 UTC

  14 points

  4

  in reply to: johnswentworth’s comment on: Ex­am­ples of Highly Coun­ter­fac­tual Dis­cov­er­ies?

  Mendel’s Laws seem counterfactual by about ˜30 years, based on partial re-discovery taking that much time. His experiments are technically something which someone could have done basically any time in last few thousand years, having basic maths

• Jan_Kulveit 22 Apr 2024 11:43 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Eliezer Yudkowsky’s comment on: GPTs are Pre­dic­tors, not Imitators

  I do agree the argument “We’re just training AIs to imitate human text, right, so that process can’t make them get any smarter than the text they’re imitating, right? So AIs shouldn’t learn abilities that humans don’t have; because why would you need those abilities to learn to imitate humans?” is wrong and clearly the answer is “Nope”.
  
  At the same time I do not think parts of your argument in the post are locally valid or good justification for the claim.
  
  Correct and locally valid argument why GPTs are not capped by human level was already written here.
  
  In a very compressed form, you can just imagine GPTs have text as their “sensory inputs” generated by the entire universe, similarly to you having your sensory inputs generated by the entire universe. Neither human intelligence nor GPTs are constrained by the complexity of the task (also: in the abstract, it’s the same task). Because of that, “task difficulty” is not a promising way how to compare these systems, and it is necessary to look into actual cognitive architectures and bounds.
  
  With the last paragraph, I’m somewhat confused by what you mean by “tasks humans evolved to solve”. Does e.g. sending humans to the Moon, or detecting Higgs boson, count as a “task humans evolved to solve” or not?

• Jan_Kulveit 18 Apr 2024 16:23 UTC

  9 points

  0

  in reply to: Ben Pace’s comment on: FHI (Fu­ture of Hu­man­ity In­sti­tute) has shut down (2005–2024)

  I sort of want to flag this interpretation of whatever gossip you heard seems misleading/​only telling small part of the story, based on my understanding.