I think the National Review is the most prestigious conservative magazine in the US, but there are various others. City Journal articles have also struck me as high-quality in the past. I think Coleman Hughes writes for them, and he did a podcast with Eliezer Yudkowsky at one point.
However, as stated in the previous link, you should likely work your way up and start by pitching lower-profile publications.
The big one probably has to do with being able to corrupt the metrics so totally that whatever you think you made them unlearn actually didn’t happen, or just being able to relearn the knowledge so fast that unlearning doesn’t matter
I favor proactive approaches to unlearning which prevent the target knowledge from being acquired in the first place. E.g. for gradient routing, if you can restrict “self-awareness and knowledge of how to corrupt metrics” to a particular submodule of the network during learning, then if that submodule isn’t active, you can be reasonably confident that the metrics aren’t currently being corrupted. (Even if that submodule sandbags and underrates its own knowledge, that should be fine if the devs know to be wary of it. Just ablate that submodule whenever you’re measuring something that matters, regardless of whether your metrics say it knows stuff!)
Unlearning techniques should probably be battle-tested in low-stakes “model organism” type contexts, where metrics corruption isn’t expected.
while I wouldn’t call it the best
Curious what areas you are most excited about!
Regarding articles which target a popular audience such as How AI Takeover Might Happen in 2 Years, I get the sense that people are preaching to the choir by posting here and on X. Is there any reason people aren’t pitching pieces like this to prestige magazines like The Atlantic or wherever else? I feel like publishing in places like that is a better way to shift the elite discourse, assuming that’s the objective. (Perhaps it’s best to pitch to publications that people in the Trump admin read?)
Here’s an article on pitching that I found on the EA Forum by searching. I assume there are lots more tips on pitching online if you search.
I think unlearning could be a good fit for automated alignment research.
Unlearning could be a very general tool to address a lot of AI threat models. It might be possible to unlearn deception, scheming, manipulation of humans, cybersecurity, etc. I challenge you to come up with an AI safety failure story that can’t, in principle, be countered through targeted unlearning in some way, shape, or form.
Relative to some other kinds of alignment research, unlearning seems easy to automate, since you can optimize metrics for how well things have been unlearned.
I like this post.
Chinas has alienated virtually all its neighbours
That sounds like an exaggeration? My impression is that China has OK/good relations with countries such as Vietnam, Cambodia, Pakistan, Indonesia, North Korea, factions in Myanmar. And Russia, of course. If you’re serious about this claim, I think you should look at a map, make a list of countries which qualify as “neighbors” based purely on geographic distance, then look up relations for each one.
What I think is more likely than EA pivoting is a handful of people launch a lifeboat and recreate a high integrity version of EA.
Thoughts on how this might be done:
Interview a bunch of people who became disillusioned. Try to identify common complaints.
For each common complaint, research organizational psychology, history of high-performing organizations, etc. and brainstorm institutional solutions to address that complaint. By “institutional solutions”, I mean approaches which claim to e.g. fix an underlying bad incentive structure, so it won’t require continuous heroic effort to address the complaint.
Combine the most promising solutions into a charter for a new association of some kind. Solicit criticism/red-teaming for the charter.
Don’t try to replace EA all at once. Start small by aiming at a particular problem present in EA, e.g. bad funding incentives, criticism (it sucks too hard to both give and receive it), or bad feedback loops in the area of AI safety. Initially focus on solving that particular problem, but also build in the capability to scale up and address additional problems if things are going well.
Don’t market this as a “replacement for EA”. There’s no reason to have an adversarial relationship. When describing the new thing, focus on the specific problem which was selected as the initial focus, plus the distinctive features of the charter and the problems they are supposed to solve.
Think of this as an experiment, where you’re aiming to test one or more theses about what charter content will cause organizational outperformance.
I think it would be interesting if someone put together a reading list on high-performing organizations, social movement history, etc. etc. I suspect this is undersupplied on the current margin, compared with observing and theorizing about EA as it exists now. Without any understanding of history, you run the risk of being a “general fighting the last war”—addressing the problems EA has now, but inadvertently introduce a new set of problems. Seems like the ideal charter would exist in the intersection of “inside view says this will fix EA’s current issues” and “outside view says this has worked well historically”.
A reading list might be too much work, but there’s really no reason not to do an LLM-enabled literature review of some kind, at the very least.
I also think a reading list for leadership could be valuable. One impression of mine is that “EA leaders” aren’t reading books about how to lead, research on leadership, or what great leaders did.
The possibility for the society-like effect of multiple power centres creating prosocial incentives on the projects
OpenAI behaves in a generally antisocial way, inconsistent with its charter, yet other power centers haven’t reined it in. Even in the EA and rationalist communities, people don’t seem to have asked questions like “Is the charter legally enforceable? Should people besides Elon Musk be suing?”
If an idea is failing in practice, it seems a bit pointless to discuss whether it will work in theory.
One idea is to use a base LLM with no RLHF, compute the perplexity of the reasoning text, and add it as an additional term in the loss function. That should help with comprehensibility, but it doesn’t necessarily help with steganography. To disincentivize steganography, you could add noise to the reasoning in various ways, and remove any incentive for terseness, to ensure the model isn’t trying to squeeze more communication into a limited token budget.
A basic idea for detecting steganography is to monitor next-token probabilities for synonym pairs. If they tend to diverge over time, that suggests the model is assigning additional hidden meaning. (This might fail if the synonym embeddings are too close.)
My preferred approach to CoT would be something like:
Give human raters the task of next-token prediction on a large text corpus. Have them write out their internal monologue when trying to predict the next word in a sentence.
Train a model to predict the internal monologue of a human rater, conditional on previous tokens.
Train a second model to predict the next token in the corpus, conditional on previous tokens in the corpus and also the written internal monologue.
Only combine the above two models in production.
Now that you’ve embedded CoT in the base model, maybe it will be powerful enough that you can discard RHLF, and replace it with some sort of fine-tuning on PhDs roleplaying as a helpful/honest/harmless chatbot.
Basically give the base model a sort of “working memory” that’s incentivized for maximal human imitativeness and interpretability. Then you could build an interface where a person can mouse over any word in a sentence and see what the model was ‘thinking’ when it chose that word. (Realistically you wouldn’t do this for every word in a sentence, just the trickier ones.)
If that’s true, perhaps the performance penalty for pinning/freezing weights in the ‘internals’, prior to the post-training, would be low. That could ease interpretability a lot, if you didn’t need to worry so much about those internals which weren’t affected by post-training?
On LessWrong, there’s a comment section where hard questions can be asked and are asked frequently.
In my experience, asking hard questions here is quite socially unrewarding. I could probably think of a dozen or so cases where I think the LW consensus “emperor” has no clothes, that I haven’t posted about, just because I expect it to be an exercise in frustration. I think I will probably quit posting here soon.
I don’t think AI policy is a good example for discourse on LessWrong. There are strategic reasons to be less transparent about how to affect public policy then for most other topics.
In terms of advocacy methods, sure. In terms of desired policies, I generally disagree.
Everything that’s written publically can be easily picked up by journalists wanting to write stories about AI.
If that’s what we are worried about, there is plenty of low-hanging fruit in terms of e.g. not tweeting wildly provocative stuff for no reason. (You can ask for examples, but be warned, sharing them might increase the probability that a journalist writes about them!)
“The far left is censorious” and “Republicans are censorious” are in no way incompatible claims :-)
Great post. Self-selection seems huge for online communities, and I think it’s no different on these fora.
Confidence level: General vague impressions and assorted thoughts follow; could very well be wrong on some details.
A disagreement I have with both the rationalist and EA communities is what the process of coming to robust conclusions looks like. In those communities, it seems like the strategy is often to identify a few super-geniuses who go do a super-deep analysis, and come to a conclusion that’s assumed to be robust and trustworthy. See the “Groupthink” section on this page for specifics.
From my perspective, I would rather see an ordinary-genius do an ordinary-depth analysis, and then have a bunch of other people ask a bunch of hard questions. If the analysis holds up against all those hard questions, then the conclusion can be taken as robust.
Everyone brings their own incentives, intuitions, and knowledge to a problem. If a single person focuses a lot on a problem, they run into diminishing returns regarding the number of angles of attack. It seems more effective to generate a lot of angles of attack by taking the union of everyone’s thoughts.
From my perspective, placing a lot of trust in top EA/LW thought leaders ironically makes them less trustworthy, because people stop asking why the emperor has no clothes.
The problem with saying the emporer has no clothes is: Either you show yourself a fool, or else you’re attacking a high-status person. Not a good prospect either way, in social terms.
EA/LW communities are an unusual niche with opaque membership norms, and people may want to retain their “insider” status. So they do extra homework before accusing the emperor of nudity, and might just procrastinate indefinitely.
There can also be a subtle aspect of circular reasoning to thought leadership: “we know this person is great because of their insights”, but also “we know this insight is great because of the person who said it”. (Certain celebrity users on these fora get 50+ positive karma on basically every top-level post. Hard to believe that the authorship isn’t coloring the perception of the content.)
A recent illustration of these principles might be the pivot to AI Pause. IIRC, it took a “super-genius” (Katja Grace) writing a super long post before Pause became popular. If an outsider simply said: “So AI is bad, why not make it illegal?”—I bet they would’ve been downvoted. And once that’s downvoted, no one feels obligated to reply. (Note, also—I don’t believe there was much reasoning transparency regarding why the pause strategy was considered unpromising at the time. You kinda had to be an insider like Katja to know the reasoning in order to critique it.)
In conclusion, I suspect there are a fair number of mistaken community beliefs which survive because (1) no “super-genius” has yet written a super-long post about them, and (2) poking around by asking hard questions is disincentivized.
Yeah, I think there are a lot of underexplored ideas along these lines.
It’s weird how so much of the internet seems locked into either the reddit model (upvotes/downvotes) or the Twitter model (likes/shares/followers), when the design space is so much larger than that. Someone like Aaron, who played such a big role in shaping the internet, seems more likely to have a gut-level belief that it can be shaped. I expect there are a lot more things like Community Notes that we could discover if we went looking for them.
I’ve always wondered what Aaron Swartz would think of the internet now, if he was still alive. He had far-left politics, but also seemed to be a big believer in openness, free speech, crowdsourcing, etc. When he was alive those were very compatible positions, and Aaron was practically the poster child for holding both of them. Nowadays the far left favors speech restrictions and is cynical about the internet.
Would Aaron have abandoned the far left, now that they are censorious? Would he have become censorious himself? Or would he have invented some clever new technology, like RSS or reddit, to try and fix the internet’s problems?
Just goes to show what a tragedy death is, I guess.
I expect escape will happen a bunch
Are you willing to name a specific year/OOM such that if there are no publicly known cases of escape by that year/OOM, you would be surprised? What, if anything, would you acknowledge as evidence that alignment is easier than you thought, here?
To ensure the definition of “escape” is not gerrymandered—do you know of any cases of escape right now? Do you think escape has already occurred and you just don’t know about it? “Escape” means something qualitatively different from any known event up to this point, yes? Does it basically refer to self-exfiltration of weights which was not requested by any human? Can we get a somewhat precise definition by any chance?
Sure there will be errors, but how important will those errors be?
Humans currently control the trajectory of humanity, and humans are error-prone. If you replace humans with something that’s error-prone in similar ways, that doesn’t seem like it’s obviously either a gain or a loss. How would such a system compare to an em of a human, for example?
If you want to show that we’re truly doomed, I think you need additional steps beyond just “there will be errors”.
Some recent-ish bird flu coverage:
Global health leader critiques ‘ineptitude’ of U.S. response to bird flu outbreak among cows
A Bird-Flu Pandemic in People? Here’s What It Might Look Like. TLDR: not good. (Reload the page and ctrl-a then ctrl-c to copy the article text before the paywall comes up.) Interesting quote: “The real danger, Dr. Lowen of Emory said, is if a farmworker becomes infected with both H5N1 and a seasonal flu virus. Flu viruses are adept at swapping genes, so a co-infection would give H5N1 opportunity to gain genes that enable it to spread among people as efficiently as seasonal flu does.”
Infectious bird flu survived milk pasteurization in lab tests, study finds. Here’s what to know.
1 in 5 milk samples from grocery stores test positive for bird flu. Why the FDA says it’s still safe to drink—see also updates from the FDA here: “Last week we announced preliminary results of a study of 297 retail dairy samples, which were all found to be negative for viable virus.” (May 10)
The FDA is making reassuring noises about pasteurized milk, but given that CDC and friends also made reassuring noises early in the COVID-19 pandemic, I’m not fully reassured.
I wonder if drinking a little bit of pasteurized milk every day would be helpful inoculation? You could hedge your bets by buying some milk from every available brand, and consuming a teaspoon from a different brand every day, gradually working up to a tablespoon etc.
About a month ago, I wrote a quick take suggesting that an early messaging mistake made by MIRI was: claim there should be a single leading FAI org, but not give specific criteria for selecting that org. That could’ve lead to a situation where Deepmind, OpenAI, and Anthropic can all think of themselves as “the best leading FAI org”.
An analogous possible mistake that’s currently being made: Claim that we should “shut it all down”, and also claim that it would be a tragedy if humanity never created AI, but not give specific criteria for when it would be appropriate to actually create AI.
What sort of specific criteria? One idea: A committee of random alignment researchers is formed to study the design; if at least X% of the committee rates the odds of success at Y% or higher, it gets the thumbs up. Not ideal criteria, just provided for the sake of illustration.
Why would this be valuable?
If we actually get a pause, it’s important to know when to unpause as well. Specific criteria could improve the odds that an unpause happens in a reasonable way.
If you want to build consensus for a pause, advertising some reasonable criteria for when we’ll unpause could get more people on board.
Don’t have time to respond in detail but a few quick clarifications/responses:
Sure, don’t feel obligated to respond, and I invite the people disagree-voting my comments to hop in as well.
— There are lots of groups focused on comms/governance. MIRI is unique only insofar as it started off as a “technical research org” and has recently pivoted more toward comms/governance.
That’s fair, when you said “pretty much any other organization in the space” I was thinking of technical orgs.
MIRI’s uniqueness does seem to suggest it has a comparative advantage for technical comms. Are there any organizations focused on that?
by MIRI’s lights, getting policymakers to understand alignment issues would be more likely to result in alignment progress than having more conversations with people in the technical alignment space
By ‘alignment progress’ do you mean an increased rate of insights per year? Due to increased alignment funding?
Anyway, I don’t think you’re going to get “shut it all down” without either a warning shot or a congressional hearing.
If you just extrapolate trends, it wouldn’t particularly surprise me to see Alex Turner at a congressional hearing arguing against “shut it all down”. Big AI has an incentive to find the best witnesses it can, and Alex Turner seems to be getting steadily more annoyed. (As am I, fwiw.)
Again, extrapolating trends, I expect MIRI’s critics like Nora Belrose will increasingly shift from the “inside game” of trying to engage w/ MIRI directly to a more “outside game” strategy of explaining to outsiders why they don’t think MIRI is credible. After the US “shuts it down”, countries like the UAE (accused of sponsoring genocide in Sudan) will likely try to quietly scoop up US AI talent. If MIRI is considered discredited in the technical community, I expect many AI researchers to accept that offer instead of retooling their career. Remember, a key mistake the board made in the OpenAI drama was underestimating the amount of leverage that individual AI researchers have, and not trying to gain mindshare with them.
Pause maximalism (by which I mean focusing 100% on getting a pause and not trying to speed alignment progress) only makes sense to me if we’re getting a ~complete ~indefinite pause. I’m not seeing a clear story for how that actually happens, absent a much broader doomer consensus. And if you’re not able to persuade your friends, you shouldn’t expect to persuade your enemies.
Right now I think MIRI only gets their stated objective in a world where we get a warning shot which creates a broader doom consensus. In that world it’s not clear advocacy makes a difference on the margin.
Likely true, but I also notice there’s been a surprising amount of drift of political opinions from the left to the right in recent years. The right tends to put their own spin on these beliefs, but I suspect many are highly influenced by the left nonetheless.
Some examples of right-coded beliefs which I suspect are, to some degree, left-inspired:
“Capitalism undermines social cohesion. Consumerization and commoditization are bad. We’re a nation, not an economy.”
“Trans women undermine women’s rights and women’s spaces. Motherhood, and women’s dignity, must be defended from neoliberal profit motives.”
“US foreign policy is controlled by a manipulative deep state that pursues unnecessary foreign interventions to benefit elites.”
“US federal institutions like the FBI are generally corrupt and need to be dismantled.”
“We can’t trust elites. They control the media. They’re out for themselves rather than ordinary Americans.”
“Your race, gender, religion, etc. are some of the most important things about you. There’s an ongoing political power struggle between e.g. different races.”
“Big tech is corrosive for society.”
“Immigration liberalization is about neoliberal billionaires undermining wages for workers like me.”
“Shrinking the size of government is not a priority. We should make sure government benefits everyday people.”
Anti-semitism, possibly.
One interesting thing has been seeing the left switch to opposing the belief when it’s adopted by the right and takes a right-coded form. E.g. US institutions are built on white supremacy and genocide, fundamentally institutionally racist, backed by illegitimate police power, and need to be defunded/decolonized/etc… but now they are being targeted by DOGE, and it’s a disaster!
(Note that the reverse shift has also happened. E.g. Trump’s approaches to economic nationalism, bilateral relations w/ China, and contempt for US institutions were all adopted by Biden by some degree.)
So yeah, my personal take is that we shouldn’t worry about publication venue that much. Just avoid insulting anyone, and make your case in a way which will appeal to the right (e.g. “we need to defend our traditional way of being human from AI”). If possible, target center-leaning publications like The Atlantic over explicitly progressive publications like Mother Jones.