Taran

Karma: 694

Taran Feb 20, 2023, 2:02 PM
3 points
0
in reply to: LawrenceC’s comment on: There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs
Also, the specific cycle attack doesn’t work against other engines I think? In the paper their adversary doesn’t transfer very well to LeelaZero, for example. So it’s more one particular AI having issues, than a fact about Go itself.
Sure, but refutations don’t transfer to different openings either, right? I feel like most game-winning insights are contingent in this sense, rather than being fundamental to the game.
EDIT: also, I think if you got arbitrary I/O access to a Magnus simulator, and then queried it millions of times in the course of doing AlphaZero style training to derive an adversarial example, I’d say it’s pretty borderline if it’s you beating beating him. Clearly there’s some level of engine skill where it’s no longer you playing!
This is a really interesting hypothetical, but I see it differently.
If the engine isn’t feeding me moves over the board (which I certainly agree would be cheating), then it has to give me something I can memorize and use later. But I can’t memorize a whole dense game tree full of winning lines (and the AI can’t calculate that anyway), so it has to give me something compressed (that is, abstract) that I can decompress and apply to a bunch of different board positions. If a human trainer did that we’d call those compressed things “insights”, “tactics”, or “strategies”, and I don’t think making the trainer into a mostly-superhuman computer changes anything. I had to learn all the tactics and insights, and I had to figure out how to apply them; what is chess, aside from that?
Also, I wouldn’t expect that Carlsen has any flaws that a generally weak player, whether human or AI, could exploit the way the cyclic adversary exploits KataGo. It would find flaws, and win consistently, but if the pattern in its play were comprehensible at all it would be the kind of thing that you have to be a super-GM yourself to take advantage of. Maybe your intuition is different here? In limit I’d definitely agree with you: if the adversarial AI spit out something like “Play 1. f3 2. Kf2 and he’ll have a stroke and start playing randomly”, then yeah, that can’t really be called “beating Carlsen” anymore. But the key point there, to me, isn’t the strength of the trainer so much as the simplicity of the example; I’d have the same objection no matter how easy the exploit was to find.
Curious what you make of this.

Taran Feb 20, 2023, 1:21 PM
9 points
−6
in reply to: pseud’s comment on: There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs
Like I said, I feel like I hear it a lot, and in practice I don’t think it’s confusing because the games that get solved by theorists and the games that get “solved” by AIs are in such vastly different complexity regimes. Like, if you heard that Arimaa had been solved, you’d immediately know which sense was meant, right?
Having said that, the voters clearly disagree and I’m not that attached to it, so I’m going to rename the post. Can you think of a single adjective or short phrase that captures the quality that chess has, and Starcraft doesn’t, WRT AI? That’s really what I want people to take away.
If I can’t think of anything better I expect I’ll go with “There are (probably) no superhuman Go AIs yet: …”.

Taran Feb 20, 2023, 11:07 AM
3 points
1
in reply to: LawrenceC’s comment on: There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs
I see where you’re coming from, but I don’t think the exploit search they did here is fundamentally different from other kinds of computer preparation. If I were going to play chess against Magnus Carlsen I’d definitely study his games with a computer, and if that computer found a stunning refutation to an opening he liked I’d definitely play it. Should we say, then, that the computer beat Carlsen, and not me? Or leave the computer aside: if I were prepping with a friend, and my friend found the winning line, should we say my friend beat Carlsen? What if I found the line in an opening book he hadn’t read?

You win games of chess, or go, by knowing things your opponent doesn’t. Where that knowledge comes from matters to the long term health of the game, but it isn’t reflected in the match score, and I think it’s the match score that matters most here.

To my embarrassment, I have not been paying much attention to adversarial RL at all. It is clearly past time to start!

Taran Feb 20, 2023, 8:09 AM
5 points
−3
in reply to: pseud’s comment on: There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs
In a game context you’re right, of course. But I often hear AI people casually say things like “chess is solved”, meaning something like “we solved the problem of getting AIs to be superhumanly good at chess” (example). For now I think we have to stop saying that about go, and instead talk about it more like how we talk about Starcraft.

Taran Feb 20, 2023, 12:08 AM
7 points
2
in reply to: ryan_b’s comment on: There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs
Well, I’m hardly an expert, I’ve just read all the posts. Marcello summed up my thinking pretty well. I don’t think I understand how you see it yet, though. Is is that the adversary’s exploit is evidence of a natural abstraction in Go that both AIs were more-or-less able to find, because it’s expressible in the language of live groups and capturing races?
You can imagine the alternative, where the “exploit” is just the adversary making moves that seem reasonable but not optimal, but then KataGo doesn’t respond well, and eventually the adversary wins without there ever being anything a human could point to and identify as a coherent strategy.

Taran Feb 19, 2023, 9:43 PM
3 points
0
in reply to: Radford Neal’s comment on: Human beats SOTA Go AI by learning an adversarial policy
In the paper, David Wu hypothesized one other ingredient: the stones involved have to form a circle rather than a tree (that is, excluding races that involve the edge of the board). I don’t think I buy his proposed mechanism but it does seem to be true that the bait group has to be floating in order for the exploit to work.

Taran Feb 19, 2023, 12:47 PM
7 points
3
on: Human beats SOTA Go AI by learning an adversarial policy
Interesting point about the scaling hypothesis. My initial take was that this was a slightly bad sign for natural abstractions: Go has a small set of fundamental abstractions, and this attack sure makes it look like KataGo didn’t quite learn some of them (liberties and capturing races), even though it was trained on however many million games of self-play and has some customizations designed to make those specific things easier. Then again, we care about Go exactly because it resisted traditional AI for so long, so maybe those abstractions aren’t as natural in KataGo’s space as they are in mine, and some other, more generally-useful architecture would be better behaved.
Definitely we’re missing efficient lifelong learning, and it’s not at all clear how to get there from current architectures.

Taran Feb 19, 2023, 12:28 PM
3 points
0
in reply to: TekhneMakre’s comment on: Human beats SOTA Go AI by learning an adversarial policy
The comments in that post are wrong, the exploit does not rely on a technicality or specific rule variant. I explained how to do it in my post, cross-posted just now with Vanessa’s post here.

There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs

TaranFeb 19, 2023, 12:25 PM

124 points

34 comments4 min readLW link

Taran Feb 14, 2023, 6:40 PM
13 points
2
on: Important fact about how people evaluate sets of arguments
Any time you get a data point about X, you get to update both on X and on the process that generated the data point. If you get several data points in a row, then as your view of the data-generating process changes you have re-evaluate all of the data it gave it you earlier. Examples:
- If somebody gives me a strong-sounding argument for X and several weak-sounding arguments for X, I’m usually less persuaded than if I just heard a strong-sounding argument for X. The weak-sounding arguments are evidence that the person I’m talking to can’t evaluate arguments well, so it’s relatively more likely that the strong-sounding argument has a flaw that I just haven’t spotted.
- If somebody gives me a strong-sounding argument for X and several reasonable-but-not-as-strong arguments against X, I’m more persuaded than just by the strong argument for X. This is because the arguments against X are evidence that the data-generating process isn’t filtered (there’s an old Zack_M_Davis post about this but I can’t find it). But this only works to the extent that the arguments against X seem like real arguments and not strawmen: weak-enough arguments against X make me less persuaded again, because they’re evidence of a deceptive data-generating process.
- If I know someone wants to persuade me of X, I mostly update less on their arguments than I would if they were indifferent, because I expect them to filter and misrepresent the data (but this one is tricky: sometimes the strong arguments are hard to find, and only the enthusiasts will bother).
- If I hear many arguments for X that seem very similar I don’t update very much after the first one, since I suspect that all the arguments are secretly correlated.
- On social media the strongest evidence is often false, because false claims can be better optimized for virality. If I hear lots of different data points of similar strength, I’ll update more strongly on each individual data point.
None of this is cheap to compute; there are a bunch of subtle, clashing considerations. So if we don’t have a lot of time, should we use the sum, or the average, or what? Equivalently: what prior should we have over data-generating processes? Here’s how I think about it:
Sum: Use this when you think your data points are independent, and not filtered in any particular way—or if you think you can precisely account for conditional dependence, selection, and so on. Ideal, but sometimes impractical and too expensive to use all the time.
Max: Useful when your main concern is noise. Probably what I use the most in my ordinary life. The idea is that most of the data I get doesn’t pertain to X at all, and the data that is about X is both subject to large random distortions and probably secretly correlated in a way that I can’t quantify very well. Nevertheless, if X is true you should expect to see signs of it, here and there, and tracking the max leaves you open to that evidence without having to worry about double-updating. As a bonus, it’s very memory efficient: you only have to remember the strongest data favoring X and the strongest data disfavoring it, and can forget all the rest.
Average: What I use when I’m evaluating an attempt at persuasion from someone I don’t know well. Averaging is a lousy way to evaluate arguments but a pretty-good-for-how-cheap-it-is way to evaluate argument-generating processes. Data points that aren’t arguments probably shouldn’t ever be averaged.
Min: I don’t think this one has any legitimate use at all. Lots of data points are only very weakly about X, even when X is true.
All of these heuristics have cases where they abjectly fail, and none of them work well when your adversary is smarter than you are.

Taran Feb 3, 2023, 10:51 PM
9 points
2
on: What Are The Preconditions/Prerequisites for Asymptotic Analysis?
Strictly speaking asymptotic analysis is not very demanding: if you have a function $f (N)$ that you can bound above in the limit as a function of $N$ , you can do asymptotic analysis to it. In practice I mostly see asymptotic analysis used to evaluate counterfactuals: you have some function or process that’s well-behaved for $N$ inputs, and you want to know if it will still be well-enough behaved if you had $2 N$ inputs instead, without actually doing the experiment. You’re rendering ten characters on the screen in your video game—could you get away with rendering 20, or would you run out of graphics card memory? Your web site is serving 100 requests per second with low latency—if you suddenly had 1000 requests per second instead, would latency still be low? Would the site still be available at all? Can we make a large language model with the exact same architecture as GPT-3, but a book-sized context window? Asymptotic analysis lets you answer questions like that without having to do experiments or think very hard—so long as you understand $f$ correctly.
When I’m reviewing software designs, I do this kind of analysis a lot. There, it’s often useful to distinguish among average-case and worst-case analyses: when you’re processing 100 million records in a big data analysis job you don’t care that much about the variance in processing any individual record, but when you’re rendering frames in a video game you work hard to make sure that every single frame gets done in less than 16 milliseconds, or whatever your budget is, even if that means your code is slower on average.
This makes it sound like a computer science thing, and for me it mostly is, but you can do the same thing to any scaling process. For example, in some cultures, when toasting before drinking, it’s considered polite for each person to toast each other person, making eye contact with them, before anyone drinks for real. If you’re in a drinking party of $N$ people, how many toasts should there be and how long should we expect them to take? Well, clearly there are about $N^{2}$ pairs of people, but you can do $N / 2$ toasts in parallel, so with good coordindation you should expect the toasts to take time proportional to $N$ …
...except that usually these toasts are done in a circle, with everyone holding still, so there’s an additional hard-to-model constraint around the shared toasting space in the circle. That is to me a prototypical example of the way that asymptotic analysis can go wrong: our model of the toasting time as a function of the number of people was fine as far as it went, but it didn’t capture all the relevant parts of the environment, so we got different scaling behavior than we expected.
(The other famous use of asymptotic analysis is in hardness proofs and complexity theory, e.g. in cryptography, but those aren’t exactly “real world processes” even though cryptography is very real).

Taran Feb 3, 2023, 11:16 AM
11 points
0
in reply to: Duncan Sabien (Deactivated)’s comment on: Basics of Rationalist Discourse
Additionally, though this is small/circumstantial, I’m pretty sure your comment came up much faster than even a five-minute timer’s worth of thought would have allowed, meaning that you spent less time trying to see the thing than it would have taken me to write out a comment that would have a good chance of making it clear to a five-year-old.
Another possibility is that he did some of his thinking before he read the post he was replying to, right? On my priors that’s even likely; I think that when people post disagreement on LW it’s mostly after thinking about the thing they’re disagreeing with, and your immediate reply didn’t really add any new information for him to update on. Your inference isn’t valid.

Taran Jan 30, 2023, 4:49 PM
3 points
in reply to: Daniel Kokotajlo’s comment on: Taboo “Outside View”
Yeah, I agree with all of this; see my own review. My guess is that Alex_Altair is making the exact mistake you tried to warn against. But, if I’m wrong, the examples would have been clarifying.

Taran Jan 23, 2023, 7:19 PM
3 points
in reply to: Alex_Altair’s comment on: Taboo “Outside View”
When you reach for this term, take a second to consider more specifically what you mean, and considering saying that more specific thing instead.
What considerations might lead you to not say the more specific thing? Can you give a few examples of cases where it’s better to say “outside view” than to say something more specific?

Taran Jan 12, 2023, 5:39 PM
6 points
5
in reply to: clone of saturn’s comment on: Escape Velocity from Bullshit Jobs
The amount of research and development coming from twitter in the 5 years before the acquisition was already pretty much negligible
That isn’t true, but I’m making a point that’s broader than just Twitter, here. If you’re a multi-billion dollar company, and you’re paying a team 5 million a year to create 10 million a year in value, then you shouldn’t fire them. Then again, if you do fire them, probably no one outside your company will be able to tell that you made a mistake: you’re only out 5 million dollars on net, and you have billions more where that came from. If you’re an outside observer trying to guess whether it was smart to fire that team or not, then you’re stuck: you don’t know how much they cost or how much value they produced.
How long do we need to wait for lawsuits or loss of clients to cause observable consequences?
In Twitter’s case the lawsuits have already started, and so has the loss of clients. But sometimes bad decisions take a long time to make themselves felt; in a case close to my heart, Digital Equipment Corporation made some bad choices in the mid to late 80s without paying any visible price until 1991 or so. Depending on how you count, that’s a lead time of 3 to 5 years. I appreciate that that’s annoying if you want to have a hot take on Musk Twitter today, but sometimes life is like that. The worlds where the Twitter firings were smart and the worlds where the Twitter firings were dumb look pretty much the same from our perspective, so we don’t get to update much. If your prior was that half or more of Twitter jobs were bullshit then by all means stay with that, but updating to that from somewhere else on the evidence we have just isn’t valid.

Taran Jan 11, 2023, 8:48 AM
33 points
15
in reply to: clone of saturn’s comment on: Escape Velocity from Bullshit Jobs
If you fire your sales staff your company will chug along just fine, but won’t take in new clients and will eventually decline through attrition of existing accounts.
If you fire your product developers your company will chug along just fine, but you won’t be able to react to customer requests or competitors.
If you fire your legal department your company will chug along just fine, but you’ll do illegal things and lose money in lawsuits.
If your fire your researchers your company will chug along just fine, but you won’t be able to exploit any more research products.
If you fire the people who do safety compliance enforcement your company will chug along just fine, but you’ll lose more money to workplace injuries and deaths (this one doesn’t apply to Twitter but is common in warehouses).
If you outsource a part of your business instead of insourcing (like running a website on the cloud instead of owning your own data centers, or doing customer service through a call center instead of your own reps) then the company will chug along just fine, and maybe not be disadvantaged in any way, but that doesn’t mean the jobs you replaced were bullshit.
In general there are lots of roles at every company that are +EV, but aren’t on the public-facing critical path. This is especially true for ad-based companies like Twitter and Facebook, because most of the customer-facing features aren’t publicly visible (remember: if you are not paying, you’re not the customer).

Taran Dec 20, 2022, 11:41 AM
7 points
on: Taboo “Outside View”
1. This post is worthwhile and correct, with clear downstream impact. It might be the only non-AI post of 2021 that I’ve heard cited in in-person conversation—and the cite immediately improved the discussion.
2. It’s clearly written and laid out; unless you’re already an excellent technical writer, you can probably learn something by ignoring its content and studying its structure.

Taran Dec 16, 2022, 11:14 AM
3 points
in reply to: Raemon’s comment on: In Defense of Attempting Hard Things, and my story of the Leverage ecosystem
That post sounds useful, I would have liked to read it.

Taran Dec 15, 2022, 1:40 PM
1 point
in reply to: Raemon’s comment on: In Defense of Attempting Hard Things, and my story of the Leverage ecosystem
Sure, I just don’t expect that it did impact peoples’ models very much*. If I’m wrong, I hope this review or the other one will pull those people out of the woodwork to explain what they learned.
*Except about Leverage, maybe, but even there...did LW-as-a-community ever come to any kind of consensus on the Leverage questions? If Geoff comes to me and asks for money to support a research project he’s in charge of, is there a standard LW answer about whether or not I should give it to him? My sense is that the discussion fizzled out unresolved, at least on LW.

Taran Dec 14, 2022, 10:44 PM
12 points
on: In Defense of Attempting Hard Things, and my story of the Leverage ecosystem
I liked this post, but I don’t think it belongs in the review. It’s very long, it needs Zoe’s also-very-long post for context, and almost everything you’ll learn is about Leverage specifically, with few generalizable insights. There are some exceptions (“What to do when society is wrong about something?” would work as a standalone post, for example), but they’re mostly just interesting questions without any work toward a solution. I think the relatively weak engagement that it got, relative to its length and quality, reflects that: Less Wrong wasn’t up for another long discussion about Leverage, and there wasn’t anything else to talk about.
Those things aren’t flaws relative to Cathleen’s goals, I don’t think, but they make this post a poor fit for the review: a didn’t make a lot of intellectual progress, and the narrow subfield it did contribute to isn’t relevant to most people.

Taran

There are (prob­a­bly) no su­per­hu­man Go AIs: strong hu­man play­ers beat the strongest AIs

There are (probably) no superhuman Go AIs: strong human players beat the strongest AIs