Jono

• JonoMay 15, 2025, 11:08 AM

  1 point

  0

  on: Fake think­ing and real thinking

  Tangentially, I fear that the model of psychology of “human agency originates from trying to correct the world such that it fits our false beliefs about it” is correct.

  I hate systematically believing false things, but do notice that my times of greatest labor have a strong sense of “trying to correct the world towards how it should be”, where the “should be” is epistemically flavored. Meditations that aim to grok how the world is not how it should be, seem to diminish my drive to rectify the flaw and yield acceptance instead.

• JonoMay 15, 2025, 10:56 AM

  1 point

  0

  on: Fake think­ing and real thinking

  Thank you. I want to stress (you already spent some words on this) that we don’t always think with the aim to find truth, or find some particular truth.

  Thinking as how to achieve some goal looks different from thinking as how to best describe some facet of the world. When evaluating your thoughts (or when someone else evaluates yours) I think it paramount to know what motivated your reasoning in the first place.

  My tag — not for getting to think real, but for knowing whether your thought was good — is to understand the constraints the thought was under.
  Who can learn the thought?
  In what way do observers look at the thought (is there access to the direct thought or only its outcome in some form?)
  What consequences would counterfactual thoughts have had?

  The failure mode I’m trying to prevent is misidentifying any failures about your past thinking due to forgetting the context under which the thinking occurred. You might evaluate that some past thought wasn’t world-saving enough, but at the time that thought might not have been trying to be that. That’s not a flaw of the thought-generation mechanism, but of the drive which invoked that.

  More generally, “motivated reasoning” is a horrible term to describe misguided cognition. Have motives and know them (not that this post used this phrase).

• JonoApr 19, 2025, 11:03 AM

  1 point

  0

  on: AI Safety Memes Wiki

  Very cool, I’m not seeing a table of contents on aisafety.info however.

• JonoApr 13, 2025, 9:13 AM

  1 point

  0

  in reply to: Viliam’s comment on: Jono’s Shortform

  Thank you very much.
  I imagined the forecasting AI to not be smart enough to be able to simulate a tiler that knows it is being simulated by us.
  Perhaps that constraint is so large that the forecasts cannot be reliable.

• JonoApr 3, 2025, 2:09 PM

  3 points

  0

  on: Jono’s Shortform

  Does this help outer alignment?

  Goal: tile the universe with niceness, without knowing what niceness is.

  Method

  We create:
  - a bunch of formulations of what niceness is.
  - a tiling AI, that given some description of niceness, tiles the universe with it.
  - a forecasting AI, that given a formulation of niceness, a description of the tiling AI, a description of the universe and some coordinates in the universe, generates a prediction of what the part of the universe at the coordinates looks like after the tiling AI has tiled it with the formulation of niceness.

  Following that, we feed our formulations of niceness into the forecasting AI, randomly sample some coordinates and evaluate whether the resulting predictions look nice.
  From this we infer which formulations of niceness are truly nice.

  Weaknesses:
  - Can we recognize utopia by randomly sampled predictions about parts of it?
  - Our forecasting AI is magnitudes weaker than the tiling AI. Can formulations of niceness turn perverse when a smarter agent optimizes for them?

  Strengths:
  - Less need to solve the ELK problem.
  - We have multiple tries at solving outer alignment.

• JonoMar 25, 2025, 12:26 PM

  1 point

  0

  on: Finish­ing The SB-1047 Doc­u­men­tary In 6 Weeks

  I have looked around a bit and have not seen any updates since November, which estimated this be finished in early February.
  Could you give another update, or link a more recent one if it exists?

• JonoJan 10, 2025, 11:07 AM

  3 points

  0

  on: Jono’s Shortform

  P(doom) can be approximately measured.
  If reality fluid describes the territory well, we should be able to see close worlds that already died off.

  For nuclear war we have some examples.
  We can estimate the odds that the Cuban missile crisis and Petrov’s decision went badly. If we accept that luck was a huge factor in us surviving those events (or not encountering events like it), we can see how unlikely our current world is to still live.

  A high P(doom) implies that we are about to (or already did) encounter some very unlikely events that worked out suspiciously well for our survival. I don’t know how public a registry of events like this should be, but it should exist.

  Our self-reporting murderers or murder-witnesses should be extraordinarily well protected from leaks however, which in part seems like a software question.

  Yes, this seems unlikely to happen, but again if your P(doom) is high, then we are only to survive in unlikely worlds. Working on this, to me, seems dignified: a way to make those unlikely worlds a bit less unlikely.

Jono’s Shortform

• JonoJun 11, 2024, 1:38 PM

  5 points

  2

  on: What should I do? (long term plan about start­ing an AI lab)

  I don’t know if you have already, but this might be the time to take a long and hard look at the probblem and consider whether deep learning is the key to solving it.

  What is the problem?

  • reckless unilateralism? → go work for policy or chip manufacturing

  • inabillity to specify human values? → that problem looks not DL at all to me

  • powerful hackers stealing all the proto-AGIs in the next 4 years? → go cybersec

  • deception? → (why focus there? why make an AI that might deceive you in the first place?) but that’s pretty ML, though I’m not sure interp is the way to go there

  • corrigibility? → might be ML, though I’m not sure all theoretical squiggles are ironed out yet

  • OOD behavior? → probably ML

  • multi-agent dynamics? → probably ML

  At the very least you ought to have a clear output channel if you’re going to work with hazardous technology. Do you have the safety-mindset that prevents you from having you dual-use tech on the streets? You’re probably familiar with the abysmal safety /​ capabilities ratio of people working in the field, any tech that helps safety as much as capability, will therefore in practice help capability more, if you don’t distribute it carefully.

  I personally would want some organisation to step up to become the keeper of secrets. I’d want them to just go all-out on cybersec, have a web of trust and basically be the solution to the unilateralists curse. That’s not ML though.

  I think this problem has a large ML-part to it, but the problem is being tackled nearly-solely by ML people. I think whatever part of the problem can be tackled with ML, won’t necessarily benefit by having more ML people on it.

[Question] Is any­one de­vel­op­ing op­ti­mi­sa­tion-ro­bust in­ter­pretabil­ity meth­ods?

• JonoJun 8, 2024, 7:46 PM

  1 point

  0

  in reply to: Bird Concept’s comment on: Closed-Source Evaluations

  agreed

Closed-Source Evaluations

AI de­mands un­prece­dented reliability

• JonoNov 29, 2023, 9:51 AM

  4 points

  3

  in reply to: Alex_Altair’s comment on: Shal­low re­view of live agen­das in al­ign­ment & safety

  ai-plans.com aims to collect research agendas and have people comment on their strengths and vulnerabilities. The discord also occasionally hosts a critique-a-ton, where people discuss specific agendas.

• JonoOct 27, 2023, 9:47 AM

  LW: 3 AF: 6

  −2

  AF

  in reply to: TurnTrout’s comment on: AI as a sci­ence, and three ob­sta­cles to al­ign­ment strategies

  We do not know, that is the relevant problem.

  Looking at the output of a black box is insufficient. You can only know by putting the black box in power, or by deeply understanding it.
  Humans are born into a world with others in power, so we know that most humans care about each other without knowing why.
  AI has no history of demonstrating friendliness in the only circumstances where that can be provably found. We can only know in advance by way of thorough understanding.

  A strong theory about AI internals should come first. Refuting Yudkowsky’s theory about how it might go wrong is irrelevant.

• Jono 8 Sep 2023 8:28 UTC

  0 points

  0

  on: The Löbian Ob­sta­cle, And Why You Should Care

  Layman here 👋
  Iiuc we cannot trust the proof of an unaligned simulacra’s suggestion because if it is smarter than us.
  Would that be a non-issue if verifying the proof is easier than making it?
  If we can know how hard it is to verify a proof without verifying, then we can find a safe protocol for communicating with this simulacra. Is this possible?

• Jono 29 Sep 2022 15:10 UTC

  2 points

  0

  on: ex­is­ten­tial self-determination

  it might be the case that any kind of meaningful values would be reasonably encodable as answers to the question “what next set of MPIs should be instantiated?”

  What examples of (meaningless) values are not answers to “What next set of MPIs should be instantiated?”

• Jono 8 Sep 2022 16:19 UTC

  2 points

  0

  on: All AGI safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  What does our world (a decade) after the employment of a successfully aligned AGI look like?

• Jono 10 Jun 2022 6:50 UTC

  1 point

  0

  in reply to: plex’s comment on: AGI Safety FAQ /​ all-dumb-ques­tions-al­lowed thread

  Thank you plex, I was not aware of this wiki.
  The pitch is nice, I’ll incorporate it.

• Jono 10 Jun 2022 6:40 UTC

  7 points

  5

  in reply to: Carl Feynman’s comment on: AGI Safety FAQ /​ all-dumb-ques­tions-al­lowed thread

  Why do I care if the people around me care about AI risk?
  
  1. when AI is going to rule we’d like the people to somehow have some power I reckon.
  I mean creating any superintelligence is a powergrab. Making one in secret is quite hostile, shouldn’t people get a say or at least insight in what their future holds?

  2. Nobody still really knows what we’d like the superint to do. I think an ML researcher is as capable of voicing their desires for the future as an artist. The field surely can benefit from interdisciplinary approaches.

  3. As with nuclear war, I’m sure politicians will care more when the people care more. AI governance is a big point. Convincing AI devs to not make the superint seems easier when a big percentage of humanity is pressuring them not to do it.

  4. Maybe this also extends to international relations. Seeing that the people of a democratic country care about the safety, makes the ventures from that country seem more reliable.

  5. I get bummed out when nobody knows what I’m talking about.