Running Lightcone Infrastructure, which runs LessWrong and Lighthaven.space. You can reach me at habryka@lesswrong.com.
(I have signed no contracts or agreements whose existence I cannot mention, which I am mentioning here as a canary)
habryka
Karma: 43,162
I think I am, all things considered, sad about this. I think libel suits are really bad tools for limiting speech, and I declined being involved with them when some of the plaintiffs offered me to be involved on behalf of LW and Lightcone.
I do think RationalWiki is one of the better applications of the relevant law, but the law is too abuse-prone, and normalizing its use would cause much more harm than RationalWiki ever caused, that I don’t think this was the right choice by the plaintiffs. I think it would have been a big personal sacrifice for the common good to not sue despite the high likelihood of success and the high ongoing personal harm incurred from RationalWiki actions, and so I have sympathy for the people who did sue, but I do think it’s pretty bad and they overall likely still made the world worse.
I think it would be extremely bad for most LW AI Alignment content if it was no longer colocated with the rest of LessWrong. Making an intellectual scene is extremely hard. The default outcome would be that it would become a bunch of fake ML research that has nothing to do with the problem. “AI Alignment” as a field does not actually have a shared methodological foundation that causes it to make sense to all be colocated in one space. LessWrong does have a shared methodology, and so it makes sense to have a forum of that kind.
I think it could make sense to have forums or subforums for specific subfields that do have enough shared perspective to make a coherent conversation possible, but I am confident that AI Alignment/AI Safety as a field does not coherently have such a thing.
Ah, I see. I did interpret the framing around “net positive” to be largely around normal companies. It’s IMO relatively easy to be net-positive, since all you need to do is to avoid harm in expectation and help in any way whatsoever, which my guess is almost any technology startup that doesn’t accelerate things, but has reasonable people at the helm, can achieve.
When we are talking more about “how to make a safety-focused company that is substantially positive on safety?”, that is a very different question in my mind.
Promoted to curated: I don’t think this post is earth-shattering, but it’s good, short, and answers an interesting question, and does so with a reasonable methodology and curiosity. And it’s not about AI, for once, which is a nice change of pace from our curation schedule these days.
I don’t think we should have norms or a culture that requires everything everyone does to be good specifically for AI Safety. Startups are mostly good because they produce economic value and solve problems for people. Sometimes they do so in a way that helps with AI Safety, sometimes not. I think Suno has helped with AI Safety because it has allowed us to make a dope album that made the rationalist culture better. Midjourney has helped us make LW better. But mostly, they just make the world better the same way most companies in history have made the world better.
I think almost all startups are really great! I think there really is a very small set of startups that end up harmful for the world, usually by specifically making a leveraged bet on trying to create very powerful AGI, or accelerating AI-driven R&D.
Because in some sense expecting a future with both of these technologies is what distinguishes our community from the rest of the world, if you end up optimizing for profit, leaning into exactly those technology then ends up a surprisingly common thing for people to do (as its where the alpha of our community relative to the rest of the world lies), which I do think is really bad.
As a concrete example, I don’t think Elicit is making the world much worse. I think its sign is not super obvious, but I don’t have a feeling they are accelerating timelines very much, or are making takeoff more sharp, or are exhausting political coordination goodwill. Similarly Midjourney I think is good for the world, so is Suno, so are basically all the AI art startups. I do think they might drive investment into the AI sector, and that might draw them into the red, but in as much as we want to have norms, and I give people advice on what to do, working on those things feels like it could definitely be net good.
Ah, yeah, that makes sense. I’ll also edit my comment to make it clear I am talking about the “Epoch” clause, to reduce ambiguity there.
I don’t understand this sentence in that case:
The original comment referenced “Matthew/Tamay/Ege”, yet you quoted Jaime to back up this claim.
But my claim is straightforwardly about the part where it’s not about “Matthew/Tamay/Ege”, but about the part where it says “Epoch”, for which the word of the director seems like the most relevant.
I agree that additionally we could also look at the Matthew/Tamay/Ege clause. I agree that you have been openly critical in many ways, and find your actions here less surprising.
“They” is referring to Epoch as an entity, which the comment referenced directly. My guess is you just missed that?
ha ha but Epoch [...] were never really safety-focused, and certainly not bright-eyed standard-view-holding EAs, I think
Of course the views of the director of Epoch at the time are highly relevant to assessing whether Epoch as an institution was presenting itself as safety focused.
Huh, by gricean implicature it IMO clearly implies that if there was a strong case that it would increase investment, then it would be a relevant and important consideration. Why bring it up otherwise?
I am really quite confident in my read here. I agree Jaime is not being maximally explicit here, but I would gladly take bets that >80% of random readers who would listen to a conversation like this, or read a comment thread like this, would walk away thinking the author does think that whether AI scaling would increase as a result of this kind of work, is considered relevant and important by Jaime.
Thank you, that is helpful information.
I don’t undertand what it would mean for “outputs” to be corrigible, so I feel like you must be talking about internal chain of thoughts here? The output of a corrigible AI and a non-corrigibile AI is the same for almost all tasks? They both try to perform any task as well as possible, the difference is how they relate to the task and how they handle interference.
This comment suggests it was maybe a shift over the last year or two (but also emphasises that at least Jaime thinks AI risk is still serious): https://www.lesswrong.com/posts/Fhwh67eJDLeaSfHzx/jonathan-claybrough-s-shortform?commentId=X3bLKX3ASvWbkNJkH
I personally take AI risks seriously, and I think they are worth investigating and preparing for.
I have drifted towards a more skeptical position on risk in the last two years. This is due to a combination of seeing the societal reaction to AI, me participating in several risk evaluation processes, and AI unfolding more gradually than I expected 10 years ago.
Currently I am more worried about concentration in AI development and how unimproved humans will retain wealth over the very long term than I am about a violent AI takeover.
Epoch has definitely described itself as safety focused to me and others. And I don’t know man, this back and forth to me sure sounds like they were branding themselves as being safety conscious:
Ofer: Can you describe your meta process for deciding what analyses to work on and how to communicate them? Analyses about the future development of transformative AI can be extremely beneficial (including via publishing them and getting many people more informed). But getting many people more hyped about scaling up ML models, for example, can also be counterproductive. Notably, The Economist article that you linked to shows your work under the title “The blessings of scale”. (I’m not making here a claim that that particular article is net-negative; just that the meta process above is very important.)
Jaime: OBJECT LEVEL REPLY:
Our current publication policy is:
Any Epoch staff member can object when we announce intention to publish a paper or blogpost.
We then have a discussion about it. If we conclude that there is a harm and that the harm outweights the benefits we refrain from publishing.
If no consensus is reached we discuss the issue with some of our trusted partners and seek advice.
Some of our work that is not published is instead disseminated privately on a case-by-case basis
We think this policy has a good mix of being flexible and giving space for Epoch staff to raise concerns.
Zach: Out of curiosity, when you “announce intention to publish a paper or blogpost,” how often has a staff member objected in the past, and how often has that led to major changes or not publishing?
Jaime: I recall three in depth conversations about particular Epoch products. None of them led to a substantive change in publication and content.
OTOH I can think of at least three instances where we decided to not pursue projects or we edited some information out of an article guided by considerations like “we may not want to call attention about this topic”.
In general I think we are good at preempting when something might be controversial or could be presented in a less conspicuous framing, and acting on it.
As well as:
Thinking about the ways publications can be harmful is something that I wish was practiced more widely in the world, specially in the field of AI.
That being said, I believe that in EA, and in particular in AI Safety, the pendulum has swung too far—we would benefit from discussing these issues more openly.
In particular, I think that talking about AI scaling is unlikely to goad major companies to invest much more in AI (there are already huge incentives). And I think EAs and people otherwise invested in AI Safety would benefit from having access to the current best guesses of the people who spend more time thinking about the topic.
This does not exempt the responsibility for Epoch and other people working on AI Strategy to be mindful of how their work could result in harm, but I felt it was important to argue for more openness in the margin.
Jaime directly emphasizes how increasing AI investment would be a reasonable and valid complaint about Epoch’s work if it was true! Look, man, if I asked this set of question, got this set of answers, while the real answer is “Yes, we think it’s pretty likely we will use the research we developed at Epoch to launch a long-time-horizon focused RL capability company”, then I sure would feel pissed (and am pissed).
I had conversations with maybe two dozen people evaluating the work of Epoch over the past few months, as well as with Epoch staff, and they were definitely generally assumed to be safety focused (if sometimes from a worldview that is more gradual disempowerment focused). I heard concerns that the leadership didn’t really care about existential risk, but nobody I talked to felt confident in that (though maybe I missed that).
We don’t yet, but have considered it a few times. It would be quite surprising if it’s a big source of revenue, so has not been a big priority for us on those grounds, but I do think it would be cool.
You are using the Markdown editor, which many fewer users use. The instructions are correct for the WYSIWYG editor (seems fine to add a footnote explaining the different syntax for Markdown).
It already has been getting a bunch harder. I am quite confident a lot of new submissions to LW are AI-generated, but the last month or two have made distinguishing them from human writing a lot harder. I still think we are pretty good, but I don’t think we are that many months away from that breaking as well.
Promoted to curated: I quite liked this post. The basic model feels like one I’ve seen explained in a bunch of other places, but I did quite like the graphs and the pedagogical approach taken in this post, and I also think book reviews continue to be one of the best ways to introduce new ideas.
The FTX lawsuit was kind of reasonable IMO! Overall made me increase my trust in the court system for settling things related to bankruptcy.
I think there are many other institutions that are better suited to helping people navigate this kind of stuff. Google can deprioritize them in their search rankings. LLMs can provide reasonable fact-checks. A community-note like system could apply to Google Search results, or people over time switch towards platforms that provide them with community-note like systems.
Indeed, my sense is RationalWiki’s influence had already been decreasing very heavily, and the period in which people did not have antibodies against them was pretty short. And I think that period would have been even shorter if people had written up what they were doing earlier (my sense is a Tracing Woodgrain’s post on some of the core people involved was pretty helpful here).