evand

Karma: 1,370

evand May 11, 2023, 2:53 AM
3 points
0
in reply to: Thane Ruthenis’s comment on: New OpenAI Paper—Language models can explain neurons in language models
This feel reminiscent of:
If the human brain were so simple that we could understand it, we would be so simple that we couldn’t.
And while it’s a well-constructed pithy quote, I don’t think it’s true. Can a system understand itself? Can a quining computer program exist? Where is the line between being able to recite itself and understand itself?
You need a model above some threshold of capability at which it can provide useful interpretations, yes, but I don’t see any obvious reason why that threshold would move up with the size of the model under interpretation.
Agreed. A quine needs some minimum complexity and/or language / environment support, but once you have one it’s usually easy to expand it. Things could go either way, and the question is an interesting one needing investigation, not bare assertion.
And the answer might depend fairly strongly on whether you take steps to make the model interpretable or a spaghetti-code turing-tar-pit mess.

evand May 4, 2023, 12:44 AM
2 points
0
in reply to: Garrett Baker’s comment on: Formalizing the “AI x-risk is unlikely because it is ridiculous” argument
I think that sounds about right. Collecting the arguments in one place is definitely helpful, and I think they carry some weight as initial heuristics, which this post helps clarify.
But I also think the technical arguments should (mostly) screen off the heuristics; the heuristics are better for evaluating whether it’s worth paying attention to the details. By the time you’re having a long debate, it’s better to spend (at least some) time looking instead of continuing to rely on the heuristics. Rhymes with Argument Screens Off Authority. (And in both cases, only mostly screens off.)

evand May 3, 2023, 1:13 AM
2 points
0
in reply to: dr_s’s comment on: The Rocket Alignment Problem, Part 2
That’s the point. SpaceX can afford to fail at this; the decision makers know it. Eliezer can afford to fail at tweet writing and knows it. So they naturally ratchet up the difficulty of the problem until they’re working on problems that maximize their expected return (in utility, not necessarily dollars). At least approximately. And then fail sometimes.
Or, for the trapeze artist… how long do they keep practicing? Do they do the no-net route when they estimate their odds of failure are 1/100? ¹⁄₁₀,000? 1e-6? They don’t push them to zero, at some point they make a call and accept the risk and go.
Why should it be any different for an entity that can one-shot those problems? Why would they wait until they had invested enough effort to one-shot it, and then do so? When instead they could just… invest less effort, attempt it earlier, take some risk of failure, and reap a greater expected reward?
The analogy suggests that entities capable of one-shotting problem X (presumably, by putting in a lot of preparatory effort, running analysis, and so on) will do so. I don’t think that’s true.
(And I think the tweet writing problem is actually an especially strong example of this—hypercompetitive social environments absolutely produce problems calibrated to be barely-solvable and that scale with ability, assuming your capability is in line with the other participants, which I assert is the case for Eliezer. he might be smarter / better at writing tweets than most, but he’s not that far ahead.)

evand May 2, 2023, 3:03 AM
6 points
1
on: The Rocket Alignment Problem, Part 2
Perhaps I’m missing something obvious, and just continuing the misunderstanding, but...

It seems to me that if you’re the sort of thing capable of one-shotting Starship launches, you don’t just hang around doing so. You tackle harder problems. The basic Umeshism: if you’re not failing sometimes, you’re not trying hard enough problems.

Even the “existential” risk of SpaceX getting permanently and entirely shut down, or just Starship getting shut down, is much closer in magnitude to the payoff than is the case in AI risk scenarios.

Some problems are well calibrated to our difficulties, because we basically understand them and there’s a feedback loop providing at least rough calibration. AI is not such a problem, rockets are, and so the analogy is a bad analogy. The problem isn’t just one of communication, the analogy breaks for important and relevant reasons.

This is extremely true for hypercompetitive domains like writing tweets that do well.

evand May 1, 2023, 2:19 AM
17 points
4
on: How can one rationally have very high or very low probabilities of extinction in a pre-paradigmatic field?
Taboo “rationally”.
I think the question you want is more like: “how can one have well-calibrated strong probabilities?”. Or maybe “correct”. I don’t think you need the word “rationally” here, and it’s almost never helpful at the object level—it’s a tool for meta-level discussions, training habits, discussing patterns, and so on.
To answer the object-level question… well, do you have well-calibrated beliefs in other domains? Did you test that? What do you think you know about your belief calibration, and how do you think you know it?
Personally, I think you mostly get there by looking at the argument structure. You can start with “well, I don’t know anything about proposition P, so it gets a 50%”, but as soon as you start looking at the details that probability shifts. What paths lead there, what don’t? If you keep coming up with complex conjunctive arguments against, and multiple-path disjunctive arguments for, the probability rapidly goes up, and can go up quite high. And that’s true even if you don’t know much about the details of those arguments, if you have any confidence at all that the process producing those is only somewhat biased. When you do have the ability to evaluate those in detail, you can get fairly high confidence.
That said, my current way of expressing my confidence on this topic is more like “on my main line scenarios...” or “conditional on no near-term giant surprises...” or “if we keep on with business as usual...”. I like the conditional predictions a lot more, partly because I feel more confident in them and partly because conditional predictions are the correct way to provide inputs to policy decisions. Different policies have different results, even if I’m not confident in our ability to enact the good ones.

evand Apr 24, 2023, 1:45 AM
2 points
0
in reply to: Raemon’s comment on: Moderation notes re: recent Said/Duncan threads
I’m still uncertain how I feel about a lot of the details on this (and am enough of a lurker rather than poster that I suspect it’s not worth my time to figure that out / write it publicly), but I just wanted to say that I think this is an extremely good thing to include:
I will probably build something that let’s people Opt Into More Said. I think it’s fairly likely the mod team will probably generally do some more heavier handed moderation in the nearish future, and I think a reasonable countermeasure to build, to alleviate some downsides of this, is to also give authors a “let this user comment unfettered on my posts, even though the mod teams have generally restricted them in some way.”
This strikes me basically as a way to move the mod team’s role more into “setting good defaults” and less “setting the only way things work”. How much y’all should move in that direction seems an open question, as it does limit how much cultivation you can do, but it seems like a very useful tool to make use of in some cases.

evand Apr 20, 2023, 4:27 AM
2 points
0
in reply to: ponkaloupe’s comment on: What if we Align the AI and nobody cares?
i could believe this number’s within 3 orders of magnitude of truth, which is probably good enough for the point of this article
It’s not. As best I can tell it’s off by more like 4+ OOM. A very quick search suggests actual usage was maybe more like 1 GWh. Back of the envelope guess: thousands of GPUs, thousands of hours, < 1kW/GPU, a few GWh.
https://www.theregister.com/2020/11/04/gpt3_carbon_footprint_estimate/
https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-planet/
i am a little surprised if you just took it 100% at face value.
Same. That doesn’t seem to rise to the quality standards I’d expect.

evand Apr 11, 2023, 3:02 AM
4 points
0
on: Outrage and Statistics into Policy
I feel like the argumentation here is kinda misleading.
Here’s a pattern that doesn’t work very well: a tragedy catches our attention, we point to statistics to show it’s an example of a distressingly common problem, and we propose laws to address the issue.
The post promises to discuss a pattern. It’s obviously a culture-war-relevant pattern, and I can see a pretty good case for it being one where all the examples are going to be at least culture-war-adjacent. It’s an important pattern, if true, so far that seems justified and worth worrying about how to improve on.
But then the post provides one example. Is it a pattern? If so, what are the other cases, and why doesn’t the post concern itself with them? If the post is about the pattern of mistakes and how to address them, and why this is a thing to worry about in general, shouldn’t there be more like 3+ examples?
The case made that gun violence is not being addressed well, and is conflating at least two different but related problems, seems quite strong. It’s a good read, and the statistics presented make a strong case that we’re not addressing it well. I liked the argument presented, and felt better informed for having read the post (note to self: I should probably dig a bit deeper on the statistics, since the conclusion is one I’m pretty sympathetic to). But if that’s the argument the post is going to make, why isn’t the title / intro paragraph something more like “we’re not addressing the main causes of gun violence” or “mass shootings that make headlines aren’t central examples of the problem” or something like that?
I assume that the gun violence example is one case of many, and that there are both general and specific lessons to be learned. I assume it would be possible to over-generalize from one example, so looking for common features and common failures would be instructive.
Overall, it felt like the post made several good points, but that the structure was a bit of a bait and switch.

evand Apr 8, 2023, 8:13 PM
6 points
2
in reply to: Vladimir_Nesov’s comment on: LW Team is adjusting moderation policy
Something like an h-index might be better than a total.

evand Apr 8, 2023, 4:33 PM
2 points
1
in reply to: exmateriae’s comment on: All AGI Safety questions welcome (especially basic ones) [April 2023]
For an extremely brief summary of the problem, I like this from Zvi:
The core reason we so often have seen creation of things we value win over destruction is, once again, that most of the optimization pressure by strong intelligences was pointing in that directly, that it was coming from humans, and the tools weren’t applying intelligence or optimization pressure. That’s about to change.
https://thezvi.wordpress.com/2023/03/28/response-to-tyler-cowens-existential-risk-ai-and-the-inevitable-turn-in-human-history/

evand Apr 7, 2023, 1:28 AM
5 points
3
in reply to: Gerald Monroe’s comment on: Eliezer Yudkowsky’s Letter in Time Magazine
Intelligence has no upper limit, instead of diminishing sharply in relative utility
It seems to me that there is a large space of intermediate claims that I interpret the letter as falling into. Namely, that if there exists an upper limit to intelligence, or a point at which the utility diminishes enough to not be worth throwing more compute cycles at it, humans are not yet approaching that limit. Returns can diminish for a long time while still being worth pursuing.
you have NO EVIDENCE that AGI is hostile or is as capable as you claim or support for any of your claims.
“No evidence” is a very different thing from “have not yet directly observed the phenomenon in question”. There is, in fact, evidence from other observations. It has not yet raised the probability to [probability 1](https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities), but there does exist such a thing as weak evidence, or strong-but-inconclusive evidence. There is evidence for this claim, and evidence for the counterclaim; we find ourselves in the position of actually needing to look at and weigh the evidence in question.

evand Apr 2, 2023, 8:43 PM
2 points
0
in reply to: dr_s’s comment on: Policy discussions follow strong contextualizing norms
It’s always possible to fudge the numbers and decide that some values are unimportant and some are super important and lo and behold, the calculation turns in your favour! In the end it’s no better than deontology or simply saying “I think this is good”; there is no point trying to vest it with a semblance of objectivity that just isn’t there.
Is this not simply the fallacy of gray?
As saying goes, it’s easy to lie with statistics, but even easier to lie without them. Certainly you can fudge the numbers to make the result say anything, but if you show your work then the fudging gets more obvious.

evand Apr 2, 2023, 5:49 PM
2 points
0
in reply to: johnswentworth’s comment on: Shannon’s Surprising Discovery
I think you missed a follow-on edit:

”Let’s unpack what that 0.36 bits means,”

evand Mar 2, 2023, 3:03 AM
2 points
0
in reply to: M. Y. Zuo’s comment on: “Rationalist Discourse” Is Like “Physicist Motors”
Metrics are only useful for comparison if they’re accepted by a sufficient broad cross section of society. Since nearly everyone engages in discourse.
I note that “sufficiently broad” might mean something like “most of LessWrong users” or “most people attending this [set of] meetups”. Just as communication is targeted at a particular audience, discourse norms are (presumably) intended for a specific context. That context probably includes things like intended users, audience, goals, and so on. I doubt “rationalist discourse” norms will align well with “televised political debate discourse” norms any time soon.
Nonetheless, I think we can discuss, measure, and improve rationalist discourse norms; and I don’t think we should concern ourselves overly much with how well those norms would work in a presidential debate or a TV ad. I suspect there are still norms that apply very broadly, with broad agreement—but those mostly aren’t the ones we’re talking about here on LessWrong.

evand Feb 26, 2023, 5:57 PM
75 points
30
on: “Rationalist Discourse” Is Like “Physicist Motors”
“Physicist motors” makes little sense because that position won out so completely that the alternative is not readily available when we think about “motor design”. But this was not always so! For a long time, wind mills and water wheels were based on intuition.
But in fact one can apply math and physics and take a “physicist motors” approach to motor design, which we see appearing in the 18th and 19th centuries. We see huge improvements in the efficiency of things like water wheels, the invention of gas thermodynamics, steam engines, and so on, playing a major role in the industrial revolution.
The difference is that motor performance is an easy target to measure and understand, and very closely related to what we actually care about (low Goodhart susceptibility). There are a bunch of parameters—cost, efficiency, energy source, size, and so on—but the number of parameters is fairly tractable. So it was very easy for the “physicist motor designers” to produce better motors, convince their customers the motors were better, and win out in the marketplace. (And no need for them to convince anyone who had contrary financial incentives.)
But “discourse” is a much more complex target, with extremely high dimensionality, and no easy way to simply win out in the market. So showing what a better approach looks like takes a huge amount of work and care, not only to develop it, but even to show that it’s better and why.
If you want to find it, the “non-physicist motors” camp is still alive and well, living in the “free energy” niche on YouTube among other places.

evand Nov 3, 2017, 1:26 AM
2 points
in reply to: MrRobot’s comment on: Satoshi Nakamoto?
Obvious kinds of humans include:
Dead humans. (Who didn’t manage to leave the coins to their heirs.)
Cryonically preserved humans hoping to use them later. (Including an obvious specific candidate.)
Humans optimistic enough about Bitcoin to think current prices are too low. (We know Nakamoto had resources, so it seems a safe bet that they could keep living on ordinary means for now.)
And the obvious: you don’t know that all of Nakamoto’s coins fit the standard assumed profile. It’s entirely possible they intentionally mined some with the regular setup and are spending a few from that pool.

evand Oct 30, 2017, 3:18 AM
2 points
in reply to: Chris_Leong’s comment on: Inadequacy and Modesty
The advanced answer to this is to create conditional prediction markets. For example: a market for whether or not the Bank of Japan implements a policy, a market for the future GDP or inflation rate of Japan (or whatever your preferred metric is), and a conditional market for (GDP given policy) and (GDP given no policy).
Then people can make conditional bets as desired, and you can report your track record, and so on. Without a prediction market you can’t, in general, solve the problem of “how good is this prediction track record really” except by looking at it in detail and making judgment calls.

evand 20 Jun 2017 5:32 UTC
1 point
in reply to: Adam Zerner’s comment on: Scope Insensitivity
I hope you have renter’s insurance, knowledge of a couple evacuation routes, and backups for any important data and papers and such.

evand 11 Jun 2017 16:54 UTC
2 points
in reply to: chaosmage’s comment on: Bet or update: fixing the will-to-wager assumption
I’m not aware of any legal implications in the US. US gambling laws basically only apply when there is a “house” taking a cut or betting to their own advantage or similar. Bets between friends where someone wins the whole stake are permitted.

As for the shady implications… spend more time hanging out with aspiring rationalists and their ilk?

evand 8 Jun 2017 14:24 UTC
0 points
in reply to: Lumifer’s comment on: Bet or update: fixing the will-to-wager assumption
The richer structure you seek for those two coins is your distribution over their probabilities. They’re both 50% likely to come up heads, given the information you have. You should be willing to make exactly the same bets about them, assuming the person offering you the bet has no more information than you do. However, if you flip each coin once and observe the results, your new probability estimate for next flips are now different.

For example, for the second coin you might have a uniform distribution (ignorance prior) over the set of all possible probabilities. In that case, if you observe a single flip that comes up heads, your probability that the next flip will be heads is now ²⁄₃.