Thanks so much for making this!
I’m hopeful this sort of dataset will grow over time as new sources come about.
In particular, I’d nominate adding MLSN (https://www.alignmentforum.org/posts/R39tGLeETfCZJ4FoE/mlsn-4-many-new-interpretability-papers-virtual-logit) to the list of newsletters in the future.
This seems like an overly alarmist take on what is a pretty old trend of research. Six years ago there was a number of universities working on similar models for the VizDoom competition (IIRC they were won by Intel and Facebook). It seems good to track this kind of research, but IMO the conclusions here are not supported at all by the evidence presented.
Do you have suggestions for domains where you do expect one-turn debate to work well, now that you’ve got these results?
Congratulations! Can you say if there will be a board, and if so who will start on it?
Longtermist X-Risk Cases for working in Semiconductor Manufacturing
Two separate pitches for jobs/roles in semiconductor manufacturing for people who are primarily interested in x-risk reduction.
Securing Semiconductor Supply Chains
This is basically the “computer security for x-risk reduction” argument applied to semiconductor manufacturing.
Briefly restating: it seems exceedingly likely that technologies crucial to x-risks are on computers or connected to computers. Improving computer security increases the likelihood that those machines are not stolen or controlled by criminals. In general, this should make things like governance and control strategy more straightforward.
This argument also applies to making sure that there isn’t any tampering with the semiconductor supply chain. In particular, we want to make sure that the designs from the designer are not modified in ways that make it easier for outside actors to steal or control information or technology.
One of the primary complaints about working in semiconductor manufacturing for longtermist reasons is accelerating semiconductor progress. I think security work here is not nearly as much as a direct driver of progress as other roles, so I would argue this as differentially x-risk reducing.
Diversifying Semiconductor Manufacturing
This one is more controversial in mainline longtermist x-risk reduction, so I’ll try to clearly signpost the hypotheses that this is based on.
The reasoning is basically:
Right now, most prosaic AI alignment techniques require access to a lot of compute
It’s possible that some prosaic AI alignment techniques (like interpretability) will require much more compute in the future
So, right now AI alignment research is at least partially gated on access to compute, and it seems plausible this will be the case in the future
So if we want to ensure these research efforts continue to have access to compute, we basically need to make sure they have enough money to buy the compute, and that there is compute to be sold.
Normally this wouldn’t be much of an issue, as in general we can trust markets to meet demands, etc. However semiconductor manufacturing is increasingly becoming a part of international conflict strategy.
In particular, much of the compute acceleration used in AI research (including AI alignment research) is manufactured in Taiwan, which seems to be coming under increasing threats.
My argument here is that I think it is possible to increase the chances that AI alignment research labs will continue to have access to compute, even in cases of large-scale geopolitical conflict. I think this can be done in ways that end up not dramatically increasing the global semiconductor manufacturing capacity by much.
I think your explanation of legibility here is basically what I have in mind, excepting that if it’s human designed it’s potentially not all encompassing. (For example, a world model that knows very little, but knows how to search for information in a library)
I think interpretability is usually a bit more narrow, and refers to developing an understanding of an illegible system. My take is that it is not “interpretability” to understand a legible system, but maybe I’m using the term differently than others here. This is why I don’t think “interpretability” applies to systems that are designed to be always-legible. (In the second graph, “interpretability” is any research that moves us upwards)
I agree that the ability to come up with totally alien and untranslateable to humans ideas gives AGI a capabilities boost. I do think that requiring a system to only use legible cognition and reasoning is a big “alignment tax”. However I don’t think that this tax is equivalent to a strong proof that legible AGI is impossible.
I think my central point of disagreement with this comment is that I do think that it’s possible to have compact world models (or at least compact enough to matter). I think if there was a strong proof that it was not possible to have a generally intelligent agent with a compact world model (or a compact function which is able to estimate and approximate a world model), that would be an update for me.
(For the record, I think of myself as a generally intelligent agent with a compact world model)
Two Graphs for why Agent Foundations is Important (according to me)
Epistemic Signpost: These are high-level abstract reasons, and I don’t go into precise detail or gears-level models. The lack of rigor is why I’m short form-ing this.
First Graph: Agent Foundations as Aligned P2B Fixpoint
P2B (a recursive acronym for Plan to P2B Better) is a framing of agency as a recursively self-reinforcing process. It resembles an abstracted version of recursive self improvement, which also incorporates recursive empowering and recursive resource gathering. Since it’s an improvement operator we can imagine stepping, I’m going to draw an analogy to gradient descent.
Imagine a highly dimensional agency landscape. In this landscape, agents follow the P2B gradient in order to improve. This can be convergent such that two slightly different agents near each other might end up at the same point in agency space after some number of P2B updates.
Most recursive processes like these have fixed point attractors — in our gradient landscape these are local minima. For P2B these are stable points of convergence.
Instead of thinking just about the fixed point attractor, lets think about the parts of agency space that flow into a given fixed point attractor. This is like analyzing watersheds on hilly terrain — which parts of the agency space flow into which attractors.
Now we can have our graph: it’s a cartoon of the “agency landscape” with different hills/valleys flowing into different local minimum, colored by which local minimum they flow into.
Here we have a lot of different attractors in agency space, but almost all of them are unaligned, what we need to do is get the tiny aligned attractor in the corner.
However it’s basically impossible to initialize an AI at one of these attractors, the best we can do is make an agent and try to understand where in agency space they will start. Building an AGI is imprecisely placing a ball on this landscape, which will roll along the P2B gradient towards its P2B attractor.
How does this relate to Agent Foundations? I see Agent Foundations as a research agenda to write up the criterion for characterizing the basin in agent space which corresponds to the aligned attractor. With this criterion, we can try to design and build an agent, such that when it P2Bs, it does so in a way that is towards an Aligned end.
Second: Agent Foundations as designing an always-legible model
ELK (Eliciting Latent Knowledge) formalized a family of alignment problems, eventually narrowing down to the Ontology Mapping Problem. This problem is about translating between some illegible machine ontology (basically it’s internal cognition) and our human ontology (concepts and relations that a person can understand).
Instead of thinking of it as a binary, I think we can think of the ontology mapping problem as a legibility spectrum. On one end of the spectrum we have our entirely illegible bayes net prosaic machine learning system. On the other end, we have totally legible machines, possibly specified in a formal language with proofs and verification.
As a second axis I’d like to imagine development progress (this can be “how far along” we are, or maybe the capabilities or empowerment of the system). Now we can show our graph, of different paths through this legibility vs development space.
Some strategies move away from legibility and never intend to get back to it. I think these plans have us building an aligned system that we don’t understand, and possibly can’t ever understand (because it can evade understanding faster than we can develop understanding).
Many prosaic alignment strategies are about going down in legibility, and then figuring out some mechanism to go back up again in legibility space. Interpretability, ontology mapping, and other approaches fit in this frame. To me, this seems better than the previous set, but still seem skeptical to me.
Finally my favorite set of strategies are ones that start legible and endeavor to never deviate from that legibility. This is where I think Agent Foundations is in this graph. I think there’s too little work on how we can build an Aligned AGI which is legible from start-to-finish, and almost all of them seem to have a bunch of overlap with Agent Foundations.
Aside: earlier I included a threshold in legibility space that‘s the “alignment threshold” but that doesn’t seem to fit right to me, so I took it out.
Maybe useful: an analogy this post brought to mind for me: Replacing “AI” with “Animals”.
Hypothetical alien civilization, observing Early Earth and commenting on whether it poses a risk.
Doesn’t optimization nature produce non-agentic animals? It mostly does, but those aren’t the ones we’re concerned with. The risk is all concentrated in the agentic animals.
Basically every animal ever is not agentic. I’ve studied animals for my entire career and I haven’t found an agentic animal yet. That doesn’t preclude them showing up in the future. We have reasons to believe that not only are agents possible, but they are likely.
Even if agentic animals showed up, they would be vastly outnumbered by all the other animals. We believe that agency will give the agentic animals such a drastic advantage, that they will seem to take over the world in a very short amount of time.
(Etc etc)
(Possible that this is in one of the things you cite, and either I missed it or I am failing to remember it)
Hacking the Transformer Prior
Neural Network Priors
I spend a bunch of time thinking about the alignment of the neural network prior for various architectures of neural networks that we expect to see in the future.
Whatever alignment failures are highly likely under the neural network prior are probably worth a lot of research attention.
Separately, it would be good to figure out knobs/levers for changing the prior distribution to be more aligned (or produce more aligned models). This includes producing more interpretable models.
Analogy to Software Development
In general, I am able to code better if I have access to a high quality library of simple utility functions. My goal here is to sketch out how we could do this for neural network learning.
Naturally Occurring Utility Functions
One way to think about the induction circuits found in the Transformer Circuits work is that they are “learned utility functions”. I think this is the sort of thing we might want to provide the networks as part of a “hacked prior”
A Language for Writing Transformer Utility Functions
Thinking Like Transformers provides a programming language, RASP, which is able to express simple functions in terms of how they would be encoded in transformers.
Concrete Research Idea: Hacking the Transformer Prior
Use RASP (or something RASP-like) to write a bunch of utility functions (such as the induction head functions).
Train a language model where a small fraction of the neural network is initialized to your utility functions (and the rest is initialized normally).
Study how the model learns to use the programmed functions. Maybe also study how those functions change (or don’t, if they’re frozen).
Future Vision
I think this could be a way to iteratively build more and more interpretable transformers, in a loop where we:
Study transformers to see what functions they are implementing
Manually implement human-understood versions of these functions
Initialize a new transformer with all of your functions, and train it
Repeat
If we have a neural network that is eventually entirely made up of human-programmed functions, we probably have an Ontologically Transparent Machine. (AN: I intend to write more thoughts on ontologically transparent machines in the near future)
I think there’s a lot going on with your equivocating the speed prior over circuits w/ a speed prior over programs.
I think a lot of the ideas in this direction are either confused by the difference between circuit priors and program priors, or at least treating them as equivalent. Unfortunately a lot of this is vague until you start specifying the domain of model. I think specifying this more clearly will help communicating about these ideas. To start with this myself, when I talk about circuit induction, I’m talking about things that look like large randomly initialized bayes nets (or deep neural networks).
Program Induction Priors are bad: I would claim that any program induction priors (simplicity prior, speed prior, others) are almost always a bad fit for developing useful intuitions about the behavior of large random bayes net machines.
Confusion between circuit induction speed prior and simplicity prior: I think your point about double descent is wrong — in particular, the speed is largely unchanged in double descent experiments, since the *width* is the parameter being varied, and all deep neural networks of the same depth have approximately the same speed (unless you mean something weird by speed).
Circuit Simplicity: You give circuit-size and circuit-depth as examples of a “speed prior”, which seems pretty nonstandard, especially when describing it as “not the simplicity prior”.
More than Speed and Simplicity: I think there are other metrics that provide interesting priors over circuits, like likelihood under some initialization distribution. In particular, I think “likelihood under the initialization distribution” is the prior that matters most, until we develop techniques that let us “hack the prior”.
Connection to Infinite-Size Neural Networks: I think research about neural networks approaching/at the infinite limit looks a lot like physics about black holes — and similarly can tell us interesting things about dynamics we should expect. In particular, for systems optimized by gradient descent, we end up with infinitesimal/nonexistent feature learning in the limit — which is interesting because all of the sub-modules/sub-circuits we start with are all we’ll ever have! This means that even if there are “simple” or “fast” circuits, if they’re not likely under the initialization distribution, then we expect they’ll have a vanishingly small effect on the output. (One way of thinking about this is in terms of the NTK, that even if we have extremely powerfully predictive modules, their predictive power will be overwhelmed by the much more common and simple features)
Hacking the Prior: Right now we don’t have a good understanding of the behavior of partially-hand coded neural networks, but I think they could serve as a new/distinct class of models (with regards to what functions are likely under the initialization distribution). Concretely, this could look like us “hand-programming” circuits or parts of neural networks, then randomly initializing the rest, and see if during training the model learns to use those programmed functions.
Interpretability Challenges
Inspired by a friend I’ve been thinking about how to launch/run interpretability competitions, and what the costs/benefits would be.
I like this idea a lot because it cuts directly at one of the hard problems of spinning up in interpretability research as a new person. The field is difficult and the objectives are vaguely defined; it’s easy to accidentally trick yourself into seeing signal in noise, and there’s never certainty that the thing you’re looking for is actually there.
On the other hand, most of the interpretability-like interventions in models (e.g. knowledge edits/updates to transformers) make models worse and not better—they usually introduce some specific and contained deficiency (e.g. predict that the Eiffel Tower is in Rome, Italy).
So the idea for Interpretability Challenges would be to use existing methods (or possibly invent new ones) to inject concrete “things to find” inside of models, release those models as challenges, and then give prizes for finding things.
Some ways this might work:
Super simple challenge: use editing techniques like ROME to edit a model, upload to google drive, and post a challenge to lesswrong. I’d probably personally put up a couple of prizes for good writeups for solutions.
CTF (Capture the Flag): the AI Village has been interested in what sorts of AI challenges/competitions could be run in tandem with infosec conferences. I think it would be pretty straightforward to build some interpretability challenges for the next AI Village CTF, or to have a whole interpretability-only CTF by itself. This is exciting to me, because its a way to recruit more people from infosec into getting interested in AI safety (which has been a goal of mine for a while).
Dixit-rules challenge league: One of the hard problems with challenges like this is how to set the difficulty. Too hard and no one makes progress. Too easy and no one learns/grows from it. I think if there were a bunch of interested people/groups, we could do a dixit style tournament: Every group takes turns proposing a challenge, and gets the most points if exactly one other group solves it (they don’t get points if everyone solves it, or if no one solves it). This has a nice self-balancing force, and would be good if there wanted to be an ongoing group who built new challenges as new interpretability research papers were published.
Please reach out to me if you’re interested in helping with efforts like this.
My Cyberwarfare Concerns: A disorganized and incomplete list
A lot of internet infrastructure (e.g. BGP / routing) basically works because all the big players mostly cooperate. There have been minor incidents and attacks but nothing major so far. It seems likely to be the case that if a major superpower was backed into a corner, it could massively disrupt the internet, which would be bad.
Cyberwar has a lot of weird asymmetries where the largest attack surfaces are private companies (not militaries/governments). This gets weirder when private companies are multinational. (Is an attack on google an attack on ireland? USA? Neither/both?)
It’s unclear who is on whose side. The Snowden leaks showed that american intelligence was hacking american companies private fibers on american soil, and the trust still hasn’t recovered. It’s a low-trust environment out there, which seems (to me) to make conflict more likely to start, and harder to contain and extinguish once started.
There is no good international “law of war” with regards to cyberwarfare. There are some works-in-progress which have been slowly advancing, but there’s nothing like the geneva convention yet. Right now existing cyber conflicts haven’t really pushed the “what is an illegal attack” sense (in the way that land mines are “illegal” in war), and the lack of clear guidance here means that in an all-out conflict there isn’t much in the way of clear limitations.
Many cyber attacks are intentionally vague or secret in origin. Some of this is because groups are distributed and only loosely connected to national powers (e.g. via funding, blind eyes, etc) and some because it’s practically useful to have plausible deniability. This really gets in the way of any sort of “cease fire” or armistice agreements—if a country comes to the peace treaty table for a given cyber conflict, this might end up implicating them as the source of an attack.
Expanding the last point more, I’m worried that there are a lot of “ratchet-up” mechanisms for cyberwarfare, but very few “ratchet-down” mechanisms. All of these worries somewhat contribute to a situation where if the currently-low-grade-burning-cyberwar turns into more of an all-out cyberwar, we’ll have very few tools for deescalation.
Relating this to my concerns about AGI safety, I think an ‘all-out cyberwar’ (or at least a much larger scale one) is one of the primary ways to trigger an AGI weapons development program. Right now it’s not clear to me that much of weapons development budget is spent on cyberweapons (as opposed to other capabilities like SIGINT), but a large-scale cyberwar seems like a reason to invest more. The more money is spent on cyberweapons development, the more likely I think it is that an AGI weapons program is started. I’m not optimistic about the alignment or safety of an AGI weapons program.
Maybe more to come in the future but that’s it for now.
I think that the authors at least did some amount of work to distinguish the eras, but agree more work could be done.
Also I agree w/ Stella here that Turing, GPT-J, GShard, and Switch are probably better fit into the “large scale“ era.
I with more of the language alignment research folks were looking into how current proposals for aligning transformers end up working on S4 models.
(I am one of said folks so maybe hypocritical to not work on it)
In particular it seems like there’s way in which it would be more interpretable than transformers:
adjustable timescale stepping (either sub-stepping, or super-stepping time)
approximately separable state spaces/dynamics—this one is crazy conjecture—it seems like it should be possible to force the state space and dynamics into separate groups, in ways that would allow analysis of them in isolation or in relation to the rest of the model
It does seem like they’re not likely to be competitive with transformers for short-context modeling anytime soon, but if they end up being differentially alignment-friendly, then we could instead try to make them more competitive.
(In general I think it’s much easier to make an approach more competitive than it is to make it more aligned)
I work on this sort of thing at OpenAI.
I think alignment datasets are a very useful part of a portfolio approach to alignment research. Right now I think there are alignment risks/concerns for which datasets like this wouldn’t help, but also there are some that it would help for.
Datasets and benchmarks more broadly are useful for forecasting progress, but this assumes smooth/continuous progress (in general a good assumption—but also good to be wary of cases where this isn’t the case).
Some thoughts from working on generating datasets for research, and using those datasets in research:
Start by building tiny versions of the dataset yourself
It’s good to switch early to paying labelers/contractors to generate and labels—they won’t be perfect at first, so there’s a lot of iterating in clarifying instructions/feedback/etc
It’s best to gather data that you’d want to use for research right away, not for some nebulous possible future research
Getting clean benchmarks that exhibit some well-defined phenomena is useful for academics and grad students
When in doubt, BIG-Bench is a good place to submit these sorts of tiny evaluative datasets
Where possible, experiment with using models to generate more data (e.g. with few-shot or generative modeling on the data you have)
Sometimes a filter is just as good as data (a classifier that distinguishes data inside the desired distribution)
I think this is a great idea, but would be best to start super small. It sounds right now like a huge project plan, but I think it could be road-mapped into something where almost every step along the path produces some valuable input.
Given the amount of funding available from charitable sources for AI alignment research these days, I think a good thing to consider is figuring out how to make instructions for contractors to generate the data, then getting money to hire the contractors and just oversee/manage them. (As opposed to trying to get volunteers to make all the data)
I worry a little bit about this == techniques which let you hide circuits in neural networks. These “hiding techniques” are a riposte to techniques based on modularity or clusterability—techniques that explore naturally emergent patterns.[1] In a world where we use alignment techniques that rely on internal circuitry being naturally modular, trojan horse networks can avoid various alignment techniques.
I expect this to happen by default for a bunch of reasons. An easy one to point to is the “free software” + “crypto anarchist” + “fuck your oversight” + “digital privacy” cluster—which will probably argue that the government shouldn’t infringe your right to build and run neural networks that are not-aligned. Similar to how encrypting personal emails subverts attempts to limit harms of email, “encrypting neural network functions” can subvert these alignment techniques.
To me, it seems like this system ends up like a steganography equilibrium—people find new hiding techniques, and others find new techniques to discover hidden information. As long as humans are both sides of this, I expect there to be progress on both sides. In the human-vs-human version of this, I think it’s not too unevenly matched.
In cases where its human-vs-AI, I strongly expect the AI wins in the limit. This is in part why I’m optimistic about things like ELK solutions which might be better at finding “hidden modules” not just naturally occurring ones.
The more time I spend with these, the more I think the idea of naturally occuring modularity or clusterability makes sense / seems likely, which has been a positive update for me
Just copy-pasting the section
We believe that Transformative Artificial Intelligence (TAI) [Karnofsky et al., 2016] is approaching [Cotra, 2020, Grace et al., 2018], and that these systems will cause catastrophic damage if they are misaligned with human values [Fox and Shulman, 2013, Omohundro, 2008]. As such, we believe it is essential to prioritize and help facilitate technical research that ensures TAI’s values will be aligned with ours.
AI Alignment generally refers to the problem of how to ensure increasingly powerful and autonomous AI systems perform the users’ wishes faithfully and without unintended consequences. Alignment is especially critical as we approach human and superhuman levels of intelligence, as powerful optimization processes amplify small errors in goal specification into large misalignments [Goodhart, 1984, Manheim and Garrabrant, 2019, Fox and Shulman, 2013], and misalignments in this regime will result in runaway optimization processes that evade alteration or shutdown [Omohundro, 2008, Benson-Tilsen and Soares, 2016, Turner et al., 2021], posing a significant existential risk to humanity. Additionally, even if the goal is specified correctly, superhuman models may still develop deceptive subsystems that attempt to influence the real world to satisfy their objectives [Hubinger et al., 2021]. While current systems are not yet at the level where the consequences of misalignment pose an existential threat, rapid progress in the field of AI has increased the concern that the alignment problem may be seriously tested in the not-too-distant future.
Much of the alignment literature focuses on the more theoretical aspects of alignment [Demski and Garrabrant, 2020, Yudkowsky and Soares, 2018, Taylor, 2016, Garrabrant et al., 2016, Armstrong and Mindermann, 2018, Hubinger et al., 2021], abstracting away the specifics of how intelligence will be implemented, due to uncertainty over the path to TAI. However, with the recent advances in capabilities, it may no longer be the case that the path to TAI is completely unpredictable. In particular, recent increases in the capabilities of large language models (LLMs) raises the possibility that the first generation of transformatively powerful AI systems may be based on similar principles and architectures as current large language models like GPT. This has motivated a number of research groups to work on “prosaic alignment” [Christiano, 2016, Askell et al., 2021, Ouyang et al., 2021], a field of study that considers the AI alignment problem in the case of TAI being built primarily with techniques already used in modern ML. We believe that due to the speed of AI progress, there is a significant chance that this assumption is true, and, therefore, that contributing and enabling contributions to prosaic alignment research will have a large impact.
The open-source release of this model is motivated by the hope that it will allow alignment researchers who would not otherwise have access to LLMs to use them. While there are negative risks due to the potential acceleration of capabilities research, which may place further time pressure on solving the alignment problem, we believe the benefits of this release outweigh the risks of accelerating capabilities research.
It’s worth probably going through the current deep learning theories that propose parts of gears-level models, and see how they fit with this. The first one that comes to mind is the Lottery Ticket Hypothesis. It seems intuitive to me that certain tasks correspond to some “tickets” that are harder to find.
I like the taxonomy in the Viering and Loog, and it links to a bunch of other interesting approaches.
This paper shows phase transitions in data quality as opposed to data size, which is an angle I hadn’t considered before.
There’s the google paper explaining neural scaling laws that describes these two regimes that can be transitioned between: variance-limited and resolution-limited. Their theory seems to predict that behavior between the two is similar to a phase boundary.
I think also there should be a bit of a null hypothesis. It seems like there are simple functional maps where even if the internal improvement on “what matters” (e.g. feature learning) is going smoothly, our metric of performance is “sharp” in a way that hides the internal improvement until some transition when it doesnt.
Accuracy metrics seem like an example of this—where you get 1 point if the correct answer is highest probability, otherwise 0 points. It’s easy to understand why this has a sharp transition in complex domains.
Personal take: I’ve been spending more and more time thinking about modularity, and it seems like modularity in learning could drive sharp transitions (e.g. “breakthroughs”).
Decomposing Negotiating Value Alignment between multiple agents
Let’s say we want two agents to come to agreement on living with each other. This seems pretty complex to specify; they agree to take each other’s values into account (somewhat), not destroy each other (with some level of confidence), etc.
Neither initially has total dominance over the other. (This implies that neither is corrigible to the other)
A good first step for these agents is to share each’s values with the other. While this could be intractably complex—it’s probably the case that values are compact/finite and can be transmitted eventually in some form.
I think this decomposes pretty clearly into ontology transmission and value assignment.
Ontology transmission is communicating one agents ontology of the objects/concepts to another. Then value assignment is communicating the relative or comparative values to different elements in the ontology.
Sometimes I get asked by intelligent people I trust in other fields, “what’s up with AI x risk?”—and I think at least part of it unpacks to this: Why don’t more people believe in / take seriously AI x-risk?
I think that is actually a pretty reasonable question. I think two follow-ups are worthwhile and I don’t know of good citations / don’t know if they exist:
a sociological/anthropological/psychological/etc study of what’s going on in people who are familiar with the ideas/reasonings of AI x-risk, but decide not to take it seriously / don’t believe it. I expect in-depth interviews would be great here.
we should probably just write up as many obvious things ourselves up front.
The latter one I can take a stab at here. Taking the perspective of someone who might be interviewed for the former:
historically, ignoring anyone that says “the end of the world is near” has been a great heuristic
very little of the public intellectual sphere engages with the topic
the public intellectual sphere that does in engages is disproportionately meme lords
most of the writings about this are exceptionally confusing and jargon-laden
there’s no college courses on this / it doesn’t have the trappings of a legitimate field
it feels a bit like a Pascal’s mugging—at the very least i’m not really prepared to try to think about actions/events with near-infinite consequences
people have been similarly doom-y about other technologies and so far the world turned out fine
we have other existential catastrophes looming (climate change, etc) that are already well understood and scientifically supported, so our efforts are better put on that than this confusing hodge-podge
this field doesn’t seem very diverse and seems to be a bit monocultural
this field doesn’t seem to have a deep/thorough understanding of all of the ways technology is affecting people’s lives negatively today
it seems weird to care about future people when there are present people suffering
I see a lot of public disagreement about whether or not AGI is even real, which makes the risk arguments feel much less trustworthy to me
I think i’m going to stop for now, but I wish there was a nice high-quality organization of these. At the very least, having the steel-version of them seems good to have around, in part as an “epistemic hygiene” thing.