Veedrac

Karma: 1,267

Veedrac May 8, 2025, 6:55 PM
2 points
−2
in reply to: winstonBosan’s comment on: Chess—“Elo” of random play?
Consider as a near-limiting case, imagine an engine that before the game began flipped a coin. On heads, it plays the game as Stockfish. On tails, it plays the game as Worstfish. What is this engine’s ELO?
I’m short of time to go into details but this should help illustrate why one should be careful treating ELO as a well defined space rather than a local approximation that’s empirically useful for computationally-limited players.

Veedrac May 1, 2025, 7:27 AM
4 points
0
in reply to: Seth Herd’s comment on: Veedrac’s Shortform
Standard xrisk arguments generally don’t extrapolate down to systems that don’t solve tasks that require instrumental goals. I think it’s reasonable to say common LLMs don’t exhibit many instrumental goals, but they also can’t solve for long-horizon goal-directed problem solving.
Prosaic risks like biorisk evals often go further and ask, if we assume the AI systems aren’t themselves very capable at this task, can we still exhibit dangerous behaviors from them ‘in the loop’? These are legitimate and interesting questions but they are a different thing.

Veedrac May 1, 2025, 12:44 AM
4 points
0
in reply to: Seth Herd’s comment on: Veedrac’s Shortform
I think Robert Miles does excellent introductory videos for newer people, and I linked him in the HN post. My goal here was different, though, which was to give a short, affirmative argument made of only directly defensible high probability claims.
I like your spin on it, too, more than those given in the linked thread, but it’s still looser, and I think there’s value giving an argument where it’s harder to disagree with the conclusion without first disagreeing with a premise. Eg. ‘some optimists assume we just won’t make AI with goals’ directly contradicts ‘capable systems necessarily have instrumental goals’, but I’m not sure it directly contradicts a premise you used.

Veedrac Apr 30, 2025, 5:30 PM
13 points
2
on: Veedrac’s Shortform
I saw a recentish post challenging people to state a clear AI xrisk argument and was surprised at how poorly formed the arguments in the comments were despite the issues getting called out. So, if you’re like apparently most of LessWrong, here’s what I consider the primary reduced argument, copied with slight edits from an HN post I made a couple years ago:
It is plausible that future systems achieve superhuman capability; capable systems necessarily have instrumental goals; instrumental goals tend to converge; human preferences are unlikely to be preserved when other goals are heavily selected for unless intentionally preserved; we don’t know how to make AI systems encode any complex preference robustly.
I should note that having a direct argument doesn’t mean other arguments like statistical precedent, analogy to evolution, or even intuition aren’t useful. It is however good mental hygiene to track when you have short reasoning chains that don’t rely on getting analogies right, since analogies are hard^[1].
1. ^
  Complete sidenote but I find this link fascinating. I wrote ‘analogies are hard’ thinking there there ought to be a Sequences post for that, not that there is. The post I found is somehow all the more convincing for the point I was making with how Yudkowsky messes up the discussion of neural networks. Were I the kind of person to write LessWrong posts rather than just imagine what they might be if I did, a better Analogies are hard would be one of the first.
What links here?
- Anthropomorphizing AI might be good, actually by Seth Herd (May 1, 2025, 1:50 PM; 35 points)

Veedrac Apr 25, 2025, 11:58 PM
2 points
4
in reply to: less-wronger-numb89’s comment on: less-wronger-numb89′s Shortform
Ultimately you have to make a bet on your guesses of reality. If your modal guess is civilizational collapse in 2-3 years, skipping uni is hardly a disproportionate action, but at the same time it’s not going to win you much either. Personally I’d leave the uni-or-not decision to the plausible worlds where the choice matters more, and look for some higher leverage change you can make for the rest.

Veedrac Apr 16, 2025, 8:38 AM
6 points
0
in reply to: Lucius Bushnaq’s comment on: johnswentworth’s Shortform
Consider, in support: Netflix has a $418B market cap. It is inconsistent to think that a $300B valuation for OpenAI or whatever’s in the news requires replacing tens of trillions of dollars of capital before the end of the decade.
Similarly, for people wanting to argue from the other direction, who might think a low current valuation is case-closed evidence against their success chances, consider that just a year ago the same argument would have discredited how they are valued today, and a year before that would have discredited where they were a year ago, and so forth. This holds similarly for historic busts in other companies. Investor sentiment is informational but clearly isn’t definitive, else stocks would never change rapidly.

Veedrac Apr 15, 2025, 8:37 PM
1 point
0
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
That’s how I interpreted it originally; you were arguing their product org vibed fake, I was arguing your vibes were miscalibrated. I’m not sure what to say to this that I didn’t say originally.

Veedrac Apr 15, 2025, 8:11 PM
2 points
0
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
But most of your criticisms in the point you gave have ~no bearing on that? If you want to make a point about how effectively OpenAI’s research moves towards AGI you should be saying things relevant to that, not giving general malaise about their business model.
Or, I might understand ‘their business model is fake which implies a lack of competence about them broadly,’ but then I go back to the whole ‘10% of people in the entire world’ and ‘expects 12B revenue’ thing.

Veedrac Apr 15, 2025, 8:04 PM
4 points
2
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
Your very first point is, to be a little uncharitable, ‘maybe OpenAI’s whole product org is fake.’ I know you have a disclaimer here but you’re talking about a product category that didn’t exist 30 months ago that today has this one website now reportedly used by 10% of people in the entire world and that the internet is saying expects ~12B revenue this year.
If your vibes are towards investing in that class of thing being fake or ‘mostly a hype machine’ then your vibes are simply not calibrated well in this domain.

Veedrac Feb 20, 2025, 8:47 PM
2 points
0
in reply to: Davidmanheim’s comment on: Gary Marcus now saying AI can’t do things it can already do
I failed to find an example easily when checking twitter this way.

Veedrac Dec 30, 2024, 10:19 AM
8 points
0
on: By default, capital will matter more than ever after AGI
Blue Origin was started two years earlier (2000 v 2002), had much better funding for most of its history,
This claim is untrue. SpaceX has never had less money than Blue Origin. It is maybe true that Blue Origin had fewer obligations attached to this money, since it was exclusively coming from Bezos, rather than a mix of investment, development contracts, and income for SpaceX, but the baseline claim that SpaceX was “money-poor” is false.

Veedrac Dec 21, 2024, 8:36 AM
5 points
0
on: Elon Musk and Solar Futurism
I need to remake the graph with more recent data, but here is a graphic of US energy additions.

https://live.staticflickr.com/65535/53977597462_2095add298_k.jpg

Nonrenewables are a walking dead at this point. I wouldn’t personally tell the story of it through Musk—I think cost curves and China are a more relevant framing—but the end point is much the same.

Veedrac Dec 17, 2024, 11:45 AM
2 points
0
in reply to: sunwillrise’s comment on: When do “brains beat brawn” in Chess? An experiment
LeelaKnightOdds has convincingly beaten both Awonder Liang and Anish Giri at 3+2 by large margins, and has an extremely strong record at 5+3 against people who have challenged it.

I think 15+0 and probably also 10+0 would be a relatively easy win for Magnus based on Awonder, a ~150 elo weaker player, taking two draws at 8+3 and a win and a draw at 10+5. At 5+3 I’m not sure because we have so little data at winnable time controls, but wouldn’t expect an easy win for either player.

It’s also certainly not the case that these few-months-old networks running a somewhat improper algorithm are the best we could build—it’s known at minimum that this Leela is tactically weaker than normal and can drop endgame wins, even if humans rarely capitalize on that.

Veedrac Nov 28, 2024, 2:54 AM
2 points
0
in reply to: NiklasGregorLessWrong’s comment on: Optimality is the tiger, and agents are its teeth
Fundamentally, the story was about the failure cases of trying to make capable systems that don’t share your values safe by preventing specific means by which its problem solving capabilities express themselves in scary ways. This is different to what you are getting at here, which is having those systems actually operationally share your values. A well aligned system, in the traditional ‘Friendly AI’ sense of alignment, simply won’t make the choices that the one in the story did.

Veedrac Sep 29, 2024, 4:56 AM
4 points
2
on: “Slow” takeoff is a terrible term for “maybe even faster takeoff, actually”
I was finding it a bit challenging to unpack what you’re saying here. I think, after a reread, that you’re using ‘slow’ and ‘fast’ in the way I would use ‘soon’ and ‘far away’ (aka. referring to the time it will occur from the present). Is this read about correct?

Veedrac Sep 25, 2024, 4:31 PM
8 points
3
in reply to: Yoav Ravid’s comment on: Yoav Ravid’s Shortform
If ‘Opt into Petrov Day’ was aside something other than a big red ominous button, I would think the obvious answer is that it’s a free choice and I’d be positively inclined towards it. Petrov Day is a good thing with good side effects, quite unlike launching nuclear weapons.

It is confusing to me that it is beside a big red ominous button. On the one hand, Petrov’s story is about the value of caution. To quote a top comment from an older Petrov Day,

Petrov thought the message looked legit, but noticed there were clues that it wasn’t.

On the other hand, risk-taking is good, opting in to good things is good, and if one is taking Petrov Day to mean ‘don’t take risks if they look scary’ I think one is taking an almost diametrically wrong message from the story.

All that said, for now I am going to fall for my own criticism and not press the big red ominous button around Petrov Day.

Veedrac 21 Aug 2024 19:53 UTC
11 points
16
on: the Giga Press was a mistake
I’m interested in having Musk-company articles on LessWrong if it can be done while preserving LessWrong norms. I’m a lot less interested in it if it means bringing in sarcasm, name calling, and ungrounded motive-speculation.

Veedrac 27 Jul 2024 20:57 UTC
11 points
3
in reply to: Viliam’s comment on: Universal Basic Income and Poverty

if, judging by looking at some economical numbers, poverty already doesn’t exist for centuries, why do we feel so poor

Let’s not forget that people who read LW, often highly intelligent and having well-paying jobs such as software development

This underlines what I find so incongruous about EY’s argument. I think I genuinely felt richer as a child eating free school meals in the UK but going to a nice school and whose parents owned a house than I do as an obscenely-by-my-standards wealthy person in San Francisco. I’m hearing this elaborate theory to explain why social security doesn’t work when I have lived through and seen in others clear evidence that it can and it does. If the question “why hasn’t a factor-100 increase in productivity felt like a factor-100 increase in productivity?” was levied at my childhood specifically, my response is that actually it felt like exactly that.

By the standards of low earning households my childhood was probably pretty atypical and I don’t mean to say there aren’t major systemic issues, especially given the number of people locked into bad employment, people with lives destroyed by addiction, people who struggle to navigate economic systems, people trapped in abusive or ineffectual families, etc. etc. etc. I really don’t want to present a case just based on my lived experience, even including those I know living various lives under government assistance. But equally I think someone’s lived experience of being wealthy in San Francisco and seeing drug addicts on the street is also not seeing an unbiased take of what social security does for poverty.

Veedrac 27 May 2024 0:36 UTC
4 points
0
in reply to: Thomas Kwa’s comment on: Veedrac’s Shortform
Eg. a moderately smart person asking it to do something else by trying a few prompts. We’re getting better at this for very simple properties but I still consider it unsolved there.

Veedrac 26 May 2024 19:50 UTC
11 points
4
on: Veedrac’s Shortform
Reply to https://twitter.com/krishnanrohit/status/1794804152444580213 , too long for twitter without a subscription so I threw it here, but do please treat it like a twitter comment.
rohit: Which part of [the traditional AI risk view] doesn’t seem accounted for here? I admit AI safety is a ‘big tent’ but there’s a reason they’re congregated together.
You wrote in your list,
the LLMs might start even setting bad objectives, by errors of omission or commission. this is a consequence of their innards not being the same as people (either hallucinations or just not having world model or misunderstanding the world)
In the context of traditional AI risk views, this misses the argument. Roughly the concern is instead like so:
ASI is by definition very capable of doing things (aka. selecting for outcomes), in at least all the ways collections of humans can. It is both theoretically true and observably the case in reality that when things are selected for, a bunch of other things that aren’t that are traded off, and that the stronger something is selected for, the more stuff ends up traded against, incidentally or not.
We should expect any ASI to have world-changing effects, and for those effects to trade off strongly against other things. There is a bunch of stuff we want that we don’t want traded off (eg. being alive).
The first problem is that we don’t know how to put any preferences into an AI such that it’s robust to even trivial selection pressure, not in theory, not in practice on existing models, and certainly not in ways that would apply to arbitrary systems that indirectly contain ML models but aren’t constrained by those models’ expressed preferences.
The second problem is that there are a bunch of instrumental goals (not eg. lying, but eg. continuing to have causal effect on the world) that are useful to almost all goals, and that are concrete examples of why an ASI would want to disempower humans. Aka. almost every thing that could plausibly be called an ASI will be effective at doing a thing, and the natural strategies for doing things involve not failing at them in easily-foreseeable ways.
Stuff like lying is not the key issue here. It often comes up because people say ‘why don’t we just ask the AI if it’s going to be bad’ and the answer is basically code for ‘you don’t seem to understand that we are talking about something that is trying to do a thing and is also good at it.’
Similarly for ‘we wouldn’t even know why it chooses outcomes, or how it accomplishes them’ — these are problematic because they are yet another reason to rule out simple fixes, not because they are fundamental to the issue. Like, if you understand why a bridge falls down, you can make a targeted fix and solve that problem, and if you don’t know then probably it’s a lot harder. But you can know every line of code of Stockfish (pre-NNUE) and still not have a chance against it, because Stockfish is actively selecting for outcomes and it is better at selecting them than you.
“LLMs have already lied to us” from the traditional AI risk crowd is similarly not about LLM lying being intrinsically scary, it is a yell of “even here you have no idea what you are doing, even here you have these creations you cannot control, so how in the world do you expect any of this to work when the child is smarter than you and it’s actually trying to achieve something?”