Linch

My “in­fo­haz­ards small work­ing group” Sig­nal Chat may have en­coun­tered minor leaks

• LinchMar 26, 2025, 10:45 PM

  3 points

  0

  in reply to: MichaelDickens’s comment on: Will Je­sus Christ re­turn in an elec­tion year?

  I think in an ideal world we’d have prediction markets structured around several different levels of investment risk, so that people with different levels of investment risk tolerance can make bets (and we might also observe fascinating differences if the odds diverge, eg if AGI probabilities are massively different between S&P 500 bets and T-bills bets, for example).

• LinchMar 25, 2025, 4:51 AM

  8 points

  5

  on: Will Je­sus Christ re­turn in an elec­tion year?

  I thought about this a bit more, and I’m worried that this is going to be a long-running problem for the reliability of prediction markets for low-probability events.

  Most of the problems we currently observe seem like “teething issues” that can be solved with higher liquidity, lower transaction costs, and better design (for example, by having bets denominated in S&P 500 or other stock portfolios rather than $s). But if you should understand “yes” predictions for many of those markets as an implicit bet on differing variances of time value of money in the future, it might be hard to construct a good design that gets around these issues to allow the markets to reflect true probabilities, especially for low-probability events.
  
  (I’m optimistic that it’s possible, unlike some other issues, but this one seems thornier than most).

• LinchMar 25, 2025, 4:23 AM

  4 points

  0

  in reply to: MondSemmel’s comment on: Linch’s Shortform

  I agree that Tracy does this at a level sufficient to count as “actually care about meritocracy” from my perspective. I would also consider Lee Kuan Yew to actually care a lot about meritocracy, for a more mainstream example.

  You could apply it to all endeavours, and conclude that “very few people are serious about <anything>”

  Yeah it’s a matter of degree not kind. But I do think many human endeavors pass my bar. I’m not saying people should devote 100% of their efforts to doing the optimal thing. 1-5% done non-optimally seems enough for me, and many people go about that for other activities.

  For example, many people care about making (risk-adjusted) returns on their money, and take significant steps towards doing so. For a less facetious example, I think global poverty EAs who earn-to-give or work to make mobile money more accessible count as “actually caring about poverty.”
  
  Similarly, many people say they care about climate change. What do you expect people to do if they care a lot about climate change? Maybe something like

  1. Push for climate-positive policies (including both direct governance and advocacy)

  2. Research or push for better research on climate change

  3. Work on clean energy

  4. Work on getting more nuclear energy

  5. Plant trees and work on other forms of carbon storage

  6. etc (as @Garrett Baker alluded to, someone who thinks a lot about climate change are probably going to have better ideas than me)

  We basically see all of these in practice, in significant numbers. Sure, most people who say they care about climate change don’t do any of the above (and (4) is rare, relatively speaking). But the ratio isn’t nearly as dismal as a complete skeptic about human nature would indicate.

• LinchMar 25, 2025, 4:16 AM

  6 points

  0

  in reply to: Garrett Baker’s comment on: Linch’s Shortform

  I thought about this for more than 10 minutes, though on a micro rather than macro level (scoped as “how can more competent people work on X” or “how can you hire talented people”). But yeah more like days rather than years.

  1. I think one-on-one talent scouting or funding are good options locally but are much less scalable than psychometric evaluations.

  2. More to the point, I haven’t seen people try to scale those things either. The closest might be something like TripleByte? Or headhunting companies? Certainly when I think of a typical (or 95th-99th percentile) “person who says they care a lot about meritocracy” I’m not imagining a recruiter, or someone in charge of such a firm. Are you?

• LinchMar 25, 2025, 4:13 AM

  4 points

  0

  in reply to: Garrett Baker’s comment on: Linch’s Shortform

  Makes sense! I agree that this is a valuable place to look. Though I am thinking about tests/​assessments in a broader way than you’re framing it here. Eg things that go into this meta-analysis, and improvements/​refinements/​new ideas, and not just narrow psychometric evaluations.

• LinchMar 24, 2025, 9:52 PM

  7 points

  0

  in reply to: Eric Neyman’s comment on: Will Je­sus Christ re­turn in an elec­tion year?

  How serious are they about respectability and people taking them seriously in the short term vs selfishly wanting more money and altruistically just wanting to make prediction markets more popular?

• LinchMar 24, 2025, 9:48 PM

  2 points

  0

  in reply to: Eric Neyman’s comment on: Will Je­sus Christ re­turn in an elec­tion year?

  Without assigning my own normative judgment, isn’t this just standard trader behavior/​professional ethics? It seems simple enough to justify thus:

  Two parties want to make a bet (trade). I create a platform to facilitate such a bet (trade). Both parties are better off by their own lights after such a trade. I helped them do something that makes them each happier, and make a healthy profit doing so. As long as I’m not doing something otherwise underhanded/​unethical, what’s the problem here?

  I don’t think it’s conceptually any different from e.g. offering memecoins on your crypto exchange, or (an atheist) selling religious texts on Amazon.

• LinchMar 24, 2025, 9:29 PM

  24 points

  1

  on: Linch’s Shortform

  Shower thought I had a while ago:
  
  Everybody loves a meritocracy until people realize that they’re the ones without merit. I mean you never hear someone say things like:

  I think America should be a meritocracy. Ruled by skill rather than personal characteristics or family connections. I mean, I love my son, and he has a great personality. But let’s be real: If we live in a meritocracy he’d be stuck in entry-level.

  (I framed the hypothetical this way because I want to exclude senior people very secure in their position who are performatively pushing for meritocracy by saying poor kids are excluded from corporate law or whatever).

  In my opinion, if you are serious about meritocracy, you figure out and promote objective tests of competency that a) has high test-retest reliability so you know it’s measuring something real, b) has high predictive validity for the outcome you are interested in getting, and c) has reasonably high accessibility so you know you’re drawing from a wide pool of talent.

  For the selection of government officials, the classic Chinese imperial service exam has high (a), low (b), medium (c). For selecting good actors, “Whether your parents are good actors” has maximally high (a), medium-high (b), very low (c). “Whether your startup exited successfully” has low (a), medium-high (b), low (c). The SATs have high (a), medium-low (b), very high (c).

  If you’re trying to make society more meritocratic, your number 1 priority should be the design and validation of tests of skill that push the Pareto frontier for various aspects of society, and your number 2 priority should be trying to push for greater incorporation of such tests.

  Given that ~ no one really does this, I conclude that very few people are serious about moving towards a meritocracy.

  (X-posted)^2

• LinchMar 11, 2025, 6:36 AM

  2 points

  0

  in reply to: Seth Herd’s comment on: Linch’s Shortform

  I agree being high-integrity and not lying is a good strategy in many real-world dealings. It’s also better for your soul. However I will not frame it as “being a bad liar” so much as “being honest.” Being high-integrity is often valuable, and ofc you accrue more benefits from actually being high-integrity when you’re also known as high-integrity. But these benefits mostly come from actually not lying, rather than lying and being bad at it.

• LinchMar 9, 2025, 9:08 PM

  15 points

  1

  on: Linch’s Shortform

  I’ve enjoyed playing social deduction games (mafia, werewolf, among us, avalon, blood on the clock tower, etc) for most of my adult life. I’ve become decent but never great at any of them. A couple of years ago, I wrote some comments on what I thought the biggest similarities and differences between social deduction games and incidences of deception in real life is. But recently, I decided that what I wrote before aren’t that important relative to what I now think of as the biggest difference:
  
  > If you are known as a good liar, is it generally advantageous or disadvantageous for you?
  
  In social deduction games, the answer is almost always “no.” Being a good liar is often advantageous, but if you are known as a good liar, this is almost always bad for you. People (rightfully) don’t trust what you say, you’re seen as an unreliable ally, etc. In games with more than two sides (e.g. Diplomacy), being a good liar is seen as a structural advantage for you, so other people are more likely to gang up on you early.

  Put another way, if you have the choice of being a good liar and being seen as a great liar, or being a great liar and seen as a good liar, it’s almost always advantageous for you to be the latter. Indeed, in many games it’s actually better to be a good liar who’s seen as a bad liar, than to be a great liar who’s seen as a great liar.

  In real life, the answer is much more mixed. Sometimes, part of being a good liar means never seeming like a good liar (“the best salesmen never makes you feel like they’re a salesman”).

  But frequently, being seen as a good liar is an asset than a liability. Thinking of people like Musk and Altman here, and also the more mundane examples of sociopaths and con men (“he’s a bastard, but he’s our bastard”). It’s often more advantageous to be seen as a good liar, than to actually be a good liar.

  This is (partially) because real life has many more repeated games of coordination, and people want allies (and don’t want enemies) who are capable. In comparison, individual board games are much more isolated and people are objectively more similar playing fields.

  Generalizing further from direct deception, a history blog post once posed the following question:
  Q: Is it better to have a mediocre army and a great reputation for fighting, or a great army with a mediocre reputation?

  Answer: The former is better, pretty much every time.

• LinchMar 5, 2025, 2:01 AM

  21 points

  0

  on: Linch’s Shortform

  Single examples almost never provides overwhelming evidence. They can provide strong evidence, but not overwhelming.

  Imagine someone arguing the following:
  

  1. You make a superficially compelling argument for invading Iraq

  2. A similar argument, if you squint, can be used to support invading Vietnam

  3. It was wrong to invade Vietnam

  4. Therefore, your argument can be ignored, and it provides ~0 evidence for the invasion of Iraq.

  In my opinion, 1-4 is not reasonable. I think it’s just not a good line of reasoning. Regardless of whether you’re for or against the Iraq invasion, and regardless of how bad you think the original argument 1 alluded to is, 4 just does not follow from 1-3.
  ___
  Well, I don’t know how Counting Arguments Provide No Evidence for AI Doom is different. In many ways the situation is worse:

  a. invading Iraq is more similar to invading Vietnam than overfitting is to scheming.

  b. As I understand it, the actual ML history was mixed. It wasn’t just counting arguments, many people also believed in the bias-variance tradeoff as an argument for overfitting. And in many NN models, the actual resolution was double-descent, which is a very interesting and confusing interaction where as the ratio of parameters to data points increases, the test error first falls, then rises, then falls again! So the appropriate analogy to scheming, if you take it very literally, is to imagine first you have goal generalization, than goal misgeneralization, than goal generalization again. But if you don’t know which end of the curve you’re on, it’s scarce comfort.

  Should you take the analogy very literally and directly? Probably not. But the less exact you make the analogy, the less bits you should be able to draw from it.
  
  ---

  I’m surprised that nobody else pointed out my critique in the full year since the post was published. Given that it was both popular and had critical engagement, I’m surprised that nobody else mentioned my criticism, which I think is more elementary than the sophisticated counterarguments other people provided. Perhaps I’m missing something.

  When I made my arguments verbally to friends, a common response was that they thought the original counting arguments were weak to begin with, so they didn’t mind weak counterarguments to it. But I think this is invalid. If you previously strongly believed in a theory, a single counterexample should update you massively (but not all the way to 0). If you previously had very little faith in a theory, a single counterexample shouldn’t update you much.

• LinchFeb 26, 2025, 10:04 PM

  2 points

  2

  in reply to: Jan Betley’s comment on: Emer­gent Misal­ign­ment: Nar­row fine­tun­ing can pro­duce broadly mis­al­igned LLMs

  I run a quick low-effort experiment with 50% secure code and 50% insecure code some time ago and I’m pretty sure this led to no emergent misalignment.

  Woah, I absolutely would not have predicted this given the rest of your results!

• LinchJan 2, 2025, 9:08 AM

  LW: 3 AF: 1

  1

  AF

  in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Eval­u­at­ing the his­tor­i­cal value mis­speci­fi­ca­tion argument

  I think I’m relatively optimistic that the difference between a system that “can (and will) do a very good job with human values when restricted to the text domain: vs “system that can do a very good job, unrestricted” isn’t that high. This is because I’m personally fairly skeptical about arguments along the lines of “words aren’t human thinking, words are mere shadows of human thinking” that people put out, at least when it comes to human values.
  
  (It’s definitely possible to come up with examples that illustrates the differences between all of human thinking and human-thinking-put-into-words; I agree about their existence, I disagree about their importance).

An­nounc­ing the Q1 2025 Long-Term Fu­ture Fund grant round

• LinchDec 10, 2024, 9:20 PM

  6 points

  0

  on: Scale Was All We Needed, At First

  So there was a lot of competitive pressure to keep pushing to make it work. A good chunk of the Superalignment team stayed on in the hope that they could win the race and use OpenAI’s lead to align the first AGI, but many of the safety people at OpenAI quit in June. We were left with a new alignment lab, Embedded Intent, and an OpenAI newly pruned of the people most wanting to slow it down.”

  “And that’s when we first started learning about this all?”

  “Publicly, yes. The OpenAI defectors were initially mysterious about their reasons for leaving, citing deep disagreements over company direction. But then some memos were leaked, SF scientists began talking, and all the attention of AI Twitter was focused on speculating about what happened. They pieced pretty much the full story together before long, but that didn’t matter soon. What did matter was that the AI world became convinced there was a powerful new technology inside OpenAI.”

  Yarden hesitated. “You’re saying that speculation, that summer hype, it led to the cyberattack in July?”

  “Well, we can’t say for certain,” I began. “But my hunch is yes. Governments had already been thinking seriously about AI for the better part of a year, and their national plans were becoming crystallized for better or worse. But AI lab security was nowhere near ready for that kind of heat.

  Wow. So many of the sociological predictions became true even though afaict the technical predictions are (thankfully) lagging behind.

• LinchDec 4, 2024, 6:55 PM

  2 points

  1

  in reply to: Martin Randall’s comment on: Eval­u­at­ing the his­tor­i­cal value mis­speci­fi­ca­tion argument

  Thanks, I’d be interested in @Matthew Barnett’s response.

A Qual­i­ta­tive Case for LTFF: Filling Crit­i­cal Ecosys­tem Gaps

• LinchNov 27, 2024, 9:54 PM

  4 points

  0

  in reply to: Going Durden’s comment on: Are the ma­jor­ity of your an­ces­tors farm­ers or non-farm­ers?

  Interesting! I didn’t consider that angle

• Linch 18 Nov 2024 5:29 UTC

  4 points

  0

  in reply to: Zach Stein-Perlman’s comment on: A Qual­i­ta­tive Case for LTFF: Filling Crit­i­cal Ecosys­tem Gaps

  Agreed, I was trying to succinctly convey something that I think is underrated, unfortunately going to miss some nuances.