RyanCarey

Karma: 1,628

RyanCarey Nov 14, 2021, 12:55 AM
25 points
in reply to: Viliam’s comment on: Sci-Hub sued in India
The most plausible way out seems to be for grantmakers to grant money conditionally on work being published as open source. Some grantmakers may benefit from doing this, despite losing some publication prestige, because the funded work will be read more widely, and the grantmaker will look like they are improving the scientific process. Researchers lose some prestige, but gain some funding. Not sure how well this has worked so far, but perhaps we could get to the world where this works, if we’re not already there.

RyanCarey Oct 13, 2021, 11:59 PM
27 points
0
in reply to: prevlev-anon’s comment on: Common knowledge facts about Leverage Research 1.0
It would be useful to have a clarification of these points, to know how different of an org you actually encountered, compared to the one I did when I (briefly) visited in 2014.
It is not true that people were expected to undergo training by their manager.
OK, but did you have any assurance that the information from charting was kept confidential from other Leveragers? I got the impression Geoff charted people who he raised money from, for example, so it at least raises the question whether information gleaned from debugging might be discussed with that person’s manager.
“being experimented on” was not my primary purpose in joining nor would I now describe it as a main focus of my time at Leverage.
OK, but would you agree that a primary activity of leverage was to do psych/sociology research, and a major (>=50%) methodology for that was self-experimentation?
I did not find the group to be overly focused on “its own sociology.”
OK, but would you agree that at least ~half of the group spent at least ~half of their time studying psychology and/or sociology, using the group as subjects?
The stated purpose of Leverage 1.0 was not to literally take over the US and/or global governance or “take over the world,”...OPs claim is false.
OK, but you agree that it was was to ensure “global coordination” and “the impossibility of bad governments”, per the plan, right? Do you agree that “the vibe was ‘take over the world’”, per the OP?
I did not believe or feel pressured to believe that Leverage was “the only organization with a plan that could possibly work.”
OK, but would you agree that many staff said this, even if you personally didn’t feel pressured to take the belief on?
I did not find “Geoff’s power and prowess as a leader [to be] a central theme.”
OK, but did you notice staff saying that he was one of the great theorists of our time? Or that a significant part of the hope for the organisation was to deploy adapt certain ideas of his, like connection theory, which “solved psychology” to deal with cases with multiple individuals, in order to design larger orgs, memes, etc?
Hopefully, the answers to these questions could be mostly-separated from our subjective impressions. Which might sound harsh, or resembling a cross-examination. But it seems necessary in order to figure out to what extent we can reach a shared understanding of “common knowledge facts”, at least about different moments in LR’s history (potentially also differing in our interpretations), versus the facts themselves actually being contested.

RyanCarey Oct 13, 2021, 11:36 AM
131 points
on: Zoe Curzi’s Experience with Leverage Research
Thanks for your courage, Zoe!
Personally, I’ve tried to maintain anonymity in online discussion of this topic for years. I dipped my toe into openly commenting last week, and immediately received an email that made it more difficult to maintain anonymity—I was told “Geoff has previously speculated to me that you are ‘throwaway’, the author of the 2018 basic facts post”. Firstly, I very much don’t appreciate my ability to maintain anonymity being narrowed like this. Rather, anonymity is a helpful defense in any sensitive online discussion, not least this one. But yes, throwaway/anonymoose is me—I posted anonymously so as to avoid adverse consequences from friends who got more involved than me. But I’m not throwaway2, anonymous, or BayAreaHuman—those three are bringing evidence that is independent from me at least.
I only visited Leverage for a couple months, back in 2014. One thing that resonated strongly with me about your post is that the discussion is badly confused by lack of public knowledge and strong narratives, about whether people are too harsh on Leverage, what biases one might have, and so on. This is why I think we often retreat to just stating “basic” or “common knowledge” facts; the facts cut through the spin.
Continuing in that spirit, I personally can attest that much of what you have said is true, and the rest congruent with the picture I built up there. They dogmatically viewed human nature as nearly arbitrarily changeable. Their plan was to study how to change their psychology, to turn themselves into Elon Musk type figures, to take over the world. This was going to work because Geoff was a legendary theoriser, Connection Theory had “solved psychology”, and the resulting debugging tools were exceptionally powerful. People “worked” for ~80 hours a week—which demonstrated the power of their productivity coaching.
Power asymmetries and insularity were present to at least some degree. I personally didn’t encounter an NDA, or talk of “demons” etc. Nor did I get a solid impression of the psychological effects on people from that short stay, though of course there must have been some.
What’s frustrating about still hearing noisy debate on this topic, so many years later, is that Leverage being a really bad org seems overdetermined at this point. On the one hand, if I ranked MIRI, CFAR, CEA, FHI, and several startups I’ve visited, in terms of how reality-distorting they can be, Leverage would score ~9, while no other would surpass ~7. (It manages to be nontransparent and cultlike in other ways too!). While on the other hand, their productive output was… also like a 2/10? It’s indefensible. But still only a fraction of the relevant information is in the open.
As you say, it’ll take time for people to build common understanding, and to come to terms with what went down. I hope the cover you’ve offered will lead some others to feel comfortable sharing their experiences, to help advance that process.
What links here?
- Research Deprioritizing External Communication by Jeff Kaufman 🔸 (EA Forum; Oct 6, 2022, 12:20 PM; 89 points)
- Research Deprioritizing External Communication by jefftk (Oct 6, 2022, 12:20 PM; 34 points)

RyanCarey Oct 8, 2021, 4:40 PM
39 points
0
in reply to: matt’s comment on: Common knowledge facts about Leverage Research 1.0
As in, 5+ years ago, around when I’d first visited the Bay, I remember meeting up 1:1 with Geoff in a cafe. One of the things I asked, in order to understand how he thought about EA strategy, was what he would do if he wasn’t busy starting Leverage. He said he’d probably start a cult, and I don’t remember any indication that he was joking whatsoever. I’d initially drafted my comment as “he told me, unjokingly”, except that it’s a long time ago, so I don’t want to give the impression that I’m quite that certain.

RyanCarey Oct 7, 2021, 4:41 PM
33 points
0
in reply to: orthonormal’s comment on: Common knowledge facts about Leverage Research 1.0
He’s also told me, deadpan, that he would like to be starting a cult if he wasn’t running Leverage.
What links here?
- Evan_Gaensbauer's comment on Zoe Curzi’s Experience with Leverage Research by Ilverin the Stupid and Offensive (Oct 14, 2021, 11:11 PM; 4 points)

RyanCarey Oct 7, 2021, 12:33 PM
7 points
0
in reply to: Daniel Kokotajlo’s comment on: Is GPT-3 already sample-efficient?
Your comparison does a disservice to the human’s sample efficiency in two ways:
1. You’re counting diverse data in the human’s environment, but you’re not comparing their performance on diverse tasks. Human’s are obviously better than GPT3 at interactive tasks, walking around, etc. For either kind of fair comparison text data & task, or diverse data & task, the human has far superior sample efficiency.
2. “fancy learning techniques” don’t count as data. If the human can get mileage out of them, all the better for the human’s sample efficiency.
So you seem to have it backwards when you say that the comparison that everyone is making is the “bad” one.

RyanCarey Oct 7, 2021, 10:35 AM
8 points
on: Is GPT-3 already sample-efficient?
I think this becomes a lot clearer if we distinguish between total and marginal thinking. GPT-3′s total sample efficiency for predicting text is poor:
- To learn to predict text, GPT-3 has to read >1000x as much text as a human can learn in their lifetime.
- To learn to win at go, AlphaGo has to play >100x times as many games as a human could play in their lifetime.
But on-the-margin, it’s very sample efficient at learning to perform new text-related tasks:
- GPT-3 can learn to perform a new text-related task as easily as a human can.
Essentially, what’s happened is GPT-3 is a kind-of mega-analytical-engine that was really sample inefficient to train up to its current level, but that can now be trained to do additional stuff at relatively little extra cost.
Does that resolve the sense of confusion/mystery, or is there more to it that I’m missing?
What links here?
- RyanCarey's comment on Forecasting transformative AI: the “biological anchors” method in a nutshell by Holden Karnofsky (EA Forum; Oct 7, 2021, 10:36 AM; 4 points)

RyanCarey Oct 4, 2021, 10:36 AM
2 points
in reply to: habryka’s comment on: The LessWrong Team is now Lightcone Infrastructure, come work with us!
Can you clarify whether you’re talking about “30% of X” i.e. 0.3*X, or “30% off X”, i.e. 0.7*X?

RyanCarey Sep 29, 2021, 9:50 AM
2 points
in reply to: Yoav Ravid’s comment on: Problems with using approval voting to elect to a multi-individual body?
Thanks; this language and these links are very useful.

RyanCarey Sep 29, 2021, 9:50 AM
2 points
in reply to: Marcus Ogren’s comment on: Problems with using approval voting to elect to a multi-individual body?
Thanks. These algorithms seem like they would be better for passing the independence of clone alternatives criterion.

[Question] Problems with using approval voting to elect to a multi-individual body?

RyanCareySep 28, 2021, 11:03 AM

8 points

13 comments1 min readLW link

RyanCarey Aug 28, 2021, 8:33 AM
LW: 2 AF: 1
AF
in reply to: abergal’s comment on: Provide feedback on Open Philanthropy’s AI alignment RFP
I imagine you could catch useful work with i) models of AI safety, or ii) analysis of failure modes, or something, though I’m obviously biased here.

RyanCarey Aug 22, 2021, 11:03 PM
LW: 14 AF: 8
AF
on: Provide feedback on Open Philanthropy’s AI alignment RFP
The implication seems to be that this RFP is for AIS work that is especially focused on DL systems. Is there likely to be a future RFP for AIS research that applies equally well to DL and non-DL systems? Regardless of where my research lands, I imagine a lot of useful and underfunded research fits in the latter category.

RyanCarey Jul 30, 2021, 2:48 PM
4 points
in reply to: gwern’s comment on: Bring up Genius
To me, his main plausible x important claim is that performance is greatly improved by subject specialisation from age <5. The fact that many geniuses enter their fields late doesn’t falsify this, since that isn’t humdrum at all—barely one in a million kids specialise in that way. I think that people who enter a field such as CS at age 30, rather than at age 20 do have a mild disadvantage, maybe 0.5SD. So I wouldn’t be surprised if starting at age 4, rather than at age 20 gave you another 1-2SD advantage. Of course, subjects like CS do benefit a lot from basic maths and so on, but there are a lot of other things that they don’t benefit from that could easily be cut from the curriculum. If this sort of thing was true, it would quite profoundly undercut the standard EA advice to keep one’s options open.

RyanCarey Jul 5, 2021, 5:58 PM
2 points
in reply to: KaragounisZ’s comment on: The Apprentice Thread
Have you considered just doing some BTC/BTC-PERP arbitrage, or betting on politics and sports? You’d probably learn what skills they’re looking for, gain some of them, and make money while you’re at it...

RyanCarey May 14, 2021, 11:02 AM
LW: 9 AF: 4
AF
in reply to: paulfchristiano’s comment on: AMA: Paul Christiano, alignment researcher
Thanks for these thoughts about the causal agenda. I basically agree with you on the facts, though I have a more favourable interpretation of how they bear on the potential of the causal incentives agenda. I’ve paraphrased the three bullet points, and responded in reverse order:
3) Many important incentives are not captured by the approach—e.g. sometimes an agent has an incentive to influence a variable, even if that variable does not cause reward attainment.
-> Agreed. We’re starting to study “side-effect incentives” (improved name pending), which have this property. We’re still figuring out whether we should just care about the union of SE incentives and control incentives, or whether SE or when, SE incentives should be considered less dangerous. Whether the causal style of incentive analysis captures much of what we care about, I think will be borne out by applying it and alternatives to a bunch of safety problems.
2) sometimes we need more specific quantities, than just D affects A.
-> Agreed. We’ve privately discussed directional quantities like “do(D=d) causes A=a” as being more safety-relevant, and are happy to hear other ideas.
1) eliminating all control-incentives seems unrealistic
-> Strongly agree it’s infeasibile to remove CIs on all variables. My more modest goal would be to prove that for particular variables (or classes of variables) such as a shut down button, or a human’s values, we can either: 1) prove how to remove control (+ side-effect) incentives, or 2) why this is impossible, given realistic assumptions. If (2), then that theoretical case could justify allocation of resources to learning-oriented approaches.
Overall, I concede that we haven’t engaged much on safety issues in the last year. Partly, it’s that the projects have had to fit within people’s PhDs. Which will also be true this year. But having some of the framework stuff behind us, we should still be able to study safety more, and gain a sense of how addressable concerns like these are, and to what extent causal decision problems/games are a really useful ontology for AI safety.

RyanCarey May 12, 2021, 2:40 PM
13 points
in reply to: abergal’s comment on: MIRI location optimization (and related topics) discussion
I think moving to the country could possibly be justified despite harms to recruitment and the rationality community, but in the official MIRI explanations, the downsides are quite underdiscussed.

RyanCarey Jan 28, 2021, 6:30 AM
2 points
in reply to: RyanCarey’s comment on: What is going on in the world?
This list is pretty relevant too: http://culturaltribes.org/home

RyanCarey Jan 26, 2021, 10:17 AM
2 points
on: What is going on in the world?
Interesting that about half of these “narratives” or “worldviews” are suffixed with “-ism”: Malthusianism, Marxism, Georgism, effective altruism, transhumanism. But most of the (newer and less popular) rationalist narratives haven’t yet been suchly named. This would be one heuristic for finding other worldviews.
More generally, if you want people to know and contrast a lot of these worldviews, it’d be useful to name them all in 1-2 words each.

RyanCarey Jan 24, 2021, 11:32 AM
12 points
on: RyanCarey’s Shortform
Causal prediction markets.

Prediction markets (and prediction tournaments more generally) may be useful for telling us not only what will happen, but which actions will achieve our goals. One proposal for getting prediction markets to help with this is to get users to make conditional predictions. For example, we can ask the question “if Biden wins the election, GDP will be higher than if Trump wins” and use that as evidence about who to elect, and so on. But conditional predictions only predict the effect of an action if the event (e.g. who is elected) is unconfounded with the outcome (GDP). It may be that higher GDP and Biden being elected have a common cause, even if electing Biden does not increase GDP directly. One way to address this would be to have the market only pay out if Biden barely wins, or Trump barely wins, so that the confounders can be assumed to be in a similar state. Another strategy for identifying the causal effect would be to randomise. We can’t randomise the election result, but we can randomise other quantities. For instance, “we generate a number from 1-100, and audit company X if we generate 1. If we generate the number 1, how much tax evasion will we find?”. In general, in order to design action-guiding prediction markets, it may be important to draw on identification strategies from the causal inference literature.

I haven’t yet checked for existing literature on this topic. Does anyone know of any?
What links here?

RyanCarey

[Question] Prob­lems with us­ing ap­proval vot­ing to elect to a multi-in­di­vi­d­ual body?

[Question] Problems with using approval voting to elect to a multi-individual body?