A Ray

Karma: 967

A Ray Jul 5, 2022, 6:55 PM
1 point
on: Alex Ray’s Shortform
More Ideas or More Consensus?
I think one aspect you can examine about a scientific field is it’s “spread”-ness of ideas and resources.
High energy particle physics is an interesting extrema here—there’s broad agreement in the field about building higher energy accelerators, and this means there can be lots of consensus about supporting a shared collaborative high energy accelerator.
I think a feature of mature scientific fields that “more consensus” can unlock more progress. Perhaps if there had been more consensus, the otherwise ill-fated superconducting super collider would have worked out. (I don’t know if other extenuating circumstances would still prevent it.)
I think a feature of less mature scientific fields that “more ideas” (and less consensus) would unlock more progress. In this case, we’re more limited about generating and validating new good ideas. One way this looks is that there’s not a lot of confidence with what to do with large sums of research funding, and instead we think our best bet is making lots of small bets.
My field (AI alignment) is a less mature scientific field in this way, I think. We don’t have a “grand plan” for alignment, which we just need to get funding. Instead we have a fractal of philanthropic organizations empowering individual grantmakers to try to get small and early ideas off the ground with small research grants.
A couple thoughts, if this model does indeed fit:
There’s a lot more we could do to orienting as a field with “the most important problem is increasing the rate of coming up with good research ideas”. In addition to being willing to fund lots of small and early stage research, I think we could factorize and interrogate the skills and mindsets needed to do this kind of work. It’s possible that this is one of the most important meta-skills we need to improve as a field.
I also think this could be more of a priority when “field building”. When recruiting or trying to raise awareness of the field, it would be good to consider more focus or priority on places where we expect to find people who are likely to be good generators of new ideas. I think one of the ways this looks is to focus on more diverse and underrepresented groups.
Finally, at some point it seems like we’ll transition to “more mature” as a field, and it’s good to spend some time thinking about what would help that go better. Understanding the history of other fields making this transition, and trying to prepare for predicted problems/issues would be good here.

A Ray Jun 29, 2022, 5:27 PM
LW: 17 AF: 8
0
AF
on: Alex Ray’s Shortform
AGI will probably be deployed by a Moral Maze
Moral Mazes is my favorite management book ever, because instead of “how to be a good manager” it’s about “empirical observations of large-scale organizational dynamics involving management”.
I wish someone would write an updated version—a lot has changed (though a lot has stayed the same) since the research for the book was done in the early 1980s.
My take (and the author’s take) is that any company of nontrivial size begins to take on the characteristics of a moral maze. It seems to be a pretty good null hypothesis—any company saying “we aren’t/won’t become a moral maze” has a pretty huge evidential burden to cross.
I keep this point in mind when thinking about strategy around when it comes time to make deployment decisions about AGI, and deploy AGI. These decisions are going to be made within the context of a moral maze.
To me, this means that some strategies (“everyone in the company has a thorough and complete understanding of AGI risks”) will almost certainly fail. I think the only strategies that work well inside of moral mazes will work at all.
To sum up my takes here:
- basically every company eventually becomes a moral maze
- AGI deployment decisions will be made in the context of a moral maze
- understanding moral maze dynamics is important to AGI deployment strategy

A Ray Jun 25, 2022, 9:58 PM
8 points
on: AI Training Should Allow Opt-Out
(Caveat: I ran the first big code scrape and worked on the code generating models which later became codex.)
My one line response: I think opt-out is obviously useful and good and should happen.
AFAIK there are various orgs/bodies working on this but kinda blanking what/where. (In particular there’s a FOSS mailing list that’s been discussing how ML training relates to FOSS license rights that seems relevant)
Opt-out strings exist today, in an insufficient form. The most well known and well respected one is probably the big-bench canary string: https://github.com/google/BIG-bench/blob/main/docs/doc.md—but this is just intended to protect data used for evaluating text models.
Mimicking the structure to comment on each point:
Simplicity
I think simplicity is points in favor of cheapness, but not points (directly) in favor of why something “should be done”. I see this as “technical cost to implement are low”, and agree.
Competitiveness
I think this also is points in favor of cheapness, but again not why it “should be done”. I see this as “expected reduction in ML perf is small”, and agree.
Ethics
I think this makes the point that we currently don’t have settled understanding on what the ethics of various options are here. People being upset at the state of things is pretty strong evidence that it’s not settled, but seems to be less strong evidence that it’s unethical. I can’t tell the point you’re trying to make here is that “we should figure out the ethics of opt-out” (which I agree with) or that “opt-out is ethically required” (which I don’t think you’ve sufficiently supported here for me to agree with).
Risk
I see this as making the point “opt-out would (very minorly) reduce AI risk”. I think this is both well supported by the arguments and technically valid. I’m personally skeptical about the amount of protection this gets us, and am mostly optimistic in applying it to non-software domains (e.g. nanotech, gain of function, virology, etc).
A personal technical prediction I can add: I think that in the software domain, it will be inexpensive for a capable system to compose any non-allowed concepts out of allowed concepts. I think this is non-obvious to traditional ML experts. In traditional ML, removing a domain from the dataset usually robustly removes it from the model—but things like the large-scale generative models mentioned in the top of the post have generalized very well across domains. (They’re still not very capable in-domain, but are similarly not-capable in domains that didn’t exist in training.) I think this “optimism about generalization” is the root of a bunch of my skepticism about domain-restriction/data-censoring as a method of restricting model capabilities.
Precedent
I think the robots.txt example is great and basically this is the one that is most directly applicable. (Other precedents exist but IMO none are as good.) I totally agree with this precedent.
Separately, there’s a lot of precedent for people circumventing or ignoring these—and I think it’s important to look at those precedents, too!
Risk Compensation
This is an interesting point. I personally don’t weigh this highly, and feel like a lot of my intuition here is attached to gut-level stuff.
As far as I know, the literature on risk compensation is almost entirely about things that are direct personal risk to someone. I don’t know of any cases of risk compensation where the risk was indirect or otherwise largely separated from the person. (At some point of indirectness this seems to reduce more to a “principal-agent problem” than a risk-compensation problem)
What’s Missing
I think it’s easy to focus on the technical implementation costs and less on the “what happens next” costs. Figuring out the legal status of this opt-out (and possibly pushing for legislation to change this) is difficult and expensive. Figuring out standards for evaluation will be similarly hard, especially as the tech itself changes rapidly.
Personal Conclusion
I think opt-out is obviously good and useful and should be done. It think its a pretty clear positive direction for ML/AI policy and regulatory development—and also I’m optimistic that this is the sort of thing that will happen largely on its own (i.e. no drastic action is required).

A Ray Jun 16, 2022, 4:58 AM
12 points
on: Alex Ray’s Shortform
Sometimes I get asked by intelligent people I trust in other fields, “what’s up with AI x risk?”—and I think at least part of it unpacks to this: Why don’t more people believe in / take seriously AI x-risk?
I think that is actually a pretty reasonable question. I think two follow-ups are worthwhile and I don’t know of good citations / don’t know if they exist:
1. a sociological/anthropological/psychological/etc study of what’s going on in people who are familiar with the ideas/reasonings of AI x-risk, but decide not to take it seriously / don’t believe it. I expect in-depth interviews would be great here.
2. we should probably just write up as many obvious things ourselves up front.
The latter one I can take a stab at here. Taking the perspective of someone who might be interviewed for the former:
- historically, ignoring anyone that says “the end of the world is near” has been a great heuristic
- very little of the public intellectual sphere engages with the topic
- the public intellectual sphere that does in engages is disproportionately meme lords
- most of the writings about this are exceptionally confusing and jargon-laden
- there’s no college courses on this / it doesn’t have the trappings of a legitimate field
- it feels a bit like a Pascal’s mugging—at the very least i’m not really prepared to try to think about actions/events with near-infinite consequences
- people have been similarly doom-y about other technologies and so far the world turned out fine
- we have other existential catastrophes looming (climate change, etc) that are already well understood and scientifically supported, so our efforts are better put on that than this confusing hodge-podge
- this field doesn’t seem very diverse and seems to be a bit monocultural
- this field doesn’t seem to have a deep/thorough understanding of all of the ways technology is affecting people’s lives negatively today
- it seems weird to care about future people when there are present people suffering
- I see a lot of public disagreement about whether or not AGI is even real, which makes the risk arguments feel much less trustworthy to me
I think i’m going to stop for now, but I wish there was a nice high-quality organization of these. At the very least, having the steel-version of them seems good to have around, in part as an “epistemic hygiene” thing.

A Ray Jun 8, 2022, 9:54 PM
LW: 12 AF: 7
AF
on: A descriptive, not prescriptive, overview of current AI Alignment Research
Thanks so much for making this!
I’m hopeful this sort of dataset will grow over time as new sources come about.
In particular, I’d nominate adding MLSN (https://www.alignmentforum.org/posts/R39tGLeETfCZJ4FoE/mlsn-4-many-new-interpretability-papers-virtual-logit) to the list of newsletters in the future.

A Ray Jun 5, 2022, 9:50 PM
LW: 1 AF: 1
AF
on: [Linkpost] A Chinese AI optimized for killing
This seems like an overly alarmist take on what is a pretty old trend of research. Six years ago there was a number of universities working on similar models for the VizDoom competition (IIRC they were won by Intel and Facebook). It seems good to track this kind of research, but IMO the conclusions here are not supported at all by the evidence presented.

A Ray Apr 12, 2022, 11:12 PM
LW: 1 AF: 1
AF
on: A Small Negative Result on Debate
Do you have suggestions for domains where you do expect one-turn debate to work well, now that you’ve got these results?

A Ray Apr 8, 2022, 6:22 PM
6 points
on: We Are Conjecture, A New Alignment Research Startup
Congratulations! Can you say if there will be a board, and if so who will start on it?

A Ray Apr 1, 2022, 12:56 AM
7 points
on: Alex Ray’s Shortform
Longtermist X-Risk Cases for working in Semiconductor Manufacturing
Two separate pitches for jobs/roles in semiconductor manufacturing for people who are primarily interested in x-risk reduction.
Securing Semiconductor Supply Chains
This is basically the “computer security for x-risk reduction” argument applied to semiconductor manufacturing.
Briefly restating: it seems exceedingly likely that technologies crucial to x-risks are on computers or connected to computers. Improving computer security increases the likelihood that those machines are not stolen or controlled by criminals. In general, this should make things like governance and control strategy more straightforward.
This argument also applies to making sure that there isn’t any tampering with the semiconductor supply chain. In particular, we want to make sure that the designs from the designer are not modified in ways that make it easier for outside actors to steal or control information or technology.
One of the primary complaints about working in semiconductor manufacturing for longtermist reasons is accelerating semiconductor progress. I think security work here is not nearly as much as a direct driver of progress as other roles, so I would argue this as differentially x-risk reducing.
Diversifying Semiconductor Manufacturing
This one is more controversial in mainline longtermist x-risk reduction, so I’ll try to clearly signpost the hypotheses that this is based on.
The reasoning is basically:
- Right now, most prosaic AI alignment techniques require access to a lot of compute
- It’s possible that some prosaic AI alignment techniques (like interpretability) will require much more compute in the future
- So, right now AI alignment research is at least partially gated on access to compute, and it seems plausible this will be the case in the future
So if we want to ensure these research efforts continue to have access to compute, we basically need to make sure they have enough money to buy the compute, and that there is compute to be sold.
Normally this wouldn’t be much of an issue, as in general we can trust markets to meet demands, etc. However semiconductor manufacturing is increasingly becoming a part of international conflict strategy.
In particular, much of the compute acceleration used in AI research (including AI alignment research) is manufactured in Taiwan, which seems to be coming under increasing threats.
My argument here is that I think it is possible to increase the chances that AI alignment research labs will continue to have access to compute, even in cases of large-scale geopolitical conflict. I think this can be done in ways that end up not dramatically increasing the global semiconductor manufacturing capacity by much.

A Ray Mar 22, 2022, 4:47 PM
LW: 3 AF: 3
AF
in reply to: Steven Byrnes’s comment on: Alex Ray’s Shortform
I think your explanation of legibility here is basically what I have in mind, excepting that if it’s human designed it’s potentially not all encompassing. (For example, a world model that knows very little, but knows how to search for information in a library)
I think interpretability is usually a bit more narrow, and refers to developing an understanding of an illegible system. My take is that it is not “interpretability” to understand a legible system, but maybe I’m using the term differently than others here. This is why I don’t think “interpretability” applies to systems that are designed to be always-legible. (In the second graph, “interpretability” is any research that moves us upwards)
I agree that the ability to come up with totally alien and untranslateable to humans ideas gives AGI a capabilities boost. I do think that requiring a system to only use legible cognition and reasoning is a big “alignment tax”. However I don’t think that this tax is equivalent to a strong proof that legible AGI is impossible.
I think my central point of disagreement with this comment is that I do think that it’s possible to have compact world models (or at least compact enough to matter). I think if there was a strong proof that it was not possible to have a generally intelligent agent with a compact world model (or a compact function which is able to estimate and approximate a world model), that would be an update for me.
(For the record, I think of myself as a generally intelligent agent with a compact world model)

A Ray Mar 22, 2022, 5:08 AM
LW: 9 AF: 6
AF
on: Alex Ray’s Shortform
Two Graphs for why Agent Foundations is Important (according to me)
Epistemic Signpost: These are high-level abstract reasons, and I don’t go into precise detail or gears-level models. The lack of rigor is why I’m short form-ing this.
First Graph: Agent Foundations as Aligned P2B Fixpoint
P2B (a recursive acronym for Plan to P2B Better) is a framing of agency as a recursively self-reinforcing process. It resembles an abstracted version of recursive self improvement, which also incorporates recursive empowering and recursive resource gathering. Since it’s an improvement operator we can imagine stepping, I’m going to draw an analogy to gradient descent.
Imagine a highly dimensional agency landscape. In this landscape, agents follow the P2B gradient in order to improve. This can be convergent such that two slightly different agents near each other might end up at the same point in agency space after some number of P2B updates.
Most recursive processes like these have fixed point attractors — in our gradient landscape these are local minima. For P2B these are stable points of convergence.
Instead of thinking just about the fixed point attractor, lets think about the parts of agency space that flow into a given fixed point attractor. This is like analyzing watersheds on hilly terrain — which parts of the agency space flow into which attractors.
Now we can have our graph: it’s a cartoon of the “agency landscape” with different hills/valleys flowing into different local minimum, colored by which local minimum they flow into.
Here we have a lot of different attractors in agency space, but almost all of them are unaligned, what we need to do is get the tiny aligned attractor in the corner.
However it’s basically impossible to initialize an AI at one of these attractors, the best we can do is make an agent and try to understand where in agency space they will start. Building an AGI is imprecisely placing a ball on this landscape, which will roll along the P2B gradient towards its P2B attractor.
How does this relate to Agent Foundations? I see Agent Foundations as a research agenda to write up the criterion for characterizing the basin in agent space which corresponds to the aligned attractor. With this criterion, we can try to design and build an agent, such that when it P2Bs, it does so in a way that is towards an Aligned end.
Second: Agent Foundations as designing an always-legible model
ELK (Eliciting Latent Knowledge) formalized a family of alignment problems, eventually narrowing down to the Ontology Mapping Problem. This problem is about translating between some illegible machine ontology (basically it’s internal cognition) and our human ontology (concepts and relations that a person can understand).
Instead of thinking of it as a binary, I think we can think of the ontology mapping problem as a legibility spectrum. On one end of the spectrum we have our entirely illegible bayes net prosaic machine learning system. On the other end, we have totally legible machines, possibly specified in a formal language with proofs and verification.
As a second axis I’d like to imagine development progress (this can be “how far along” we are, or maybe the capabilities or empowerment of the system). Now we can show our graph, of different paths through this legibility vs development space.
Some strategies move away from legibility and never intend to get back to it. I think these plans have us building an aligned system that we don’t understand, and possibly can’t ever understand (because it can evade understanding faster than we can develop understanding).
Many prosaic alignment strategies are about going down in legibility, and then figuring out some mechanism to go back up again in legibility space. Interpretability, ontology mapping, and other approaches fit in this frame. To me, this seems better than the previous set, but still seem skeptical to me.
Finally my favorite set of strategies are ones that start legible and endeavor to never deviate from that legibility. This is where I think Agent Foundations is in this graph. I think there’s too little work on how we can build an Aligned AGI which is legible from start-to-finish, and almost all of them seem to have a bunch of overlap with Agent Foundations.
Aside: earlier I included a threshold in legibility space that‘s the “alignment threshold” but that doesn’t seem to fit right to me, so I took it out.

A Ray Mar 22, 2022, 4:04 AM
LW: 3 AF: 3
AF
on: Agency: What it is and why it matters
Maybe useful: an analogy this post brought to mind for me: Replacing “AI” with “Animals”.

Hypothetical alien civilization, observing Early Earth and commenting on whether it poses a risk.
Doesn’t ~~optimization~~ nature produce non-agentic animals? It mostly does, but those aren’t the ones we’re concerned with. The risk is all concentrated in the agentic animals.
Basically every animal ever is not agentic. I’ve studied animals for my entire career and I haven’t found an agentic animal yet. That doesn’t preclude them showing up in the future. We have reasons to believe that not only are agents possible, but they are likely.
Even if agentic animals showed up, they would be vastly outnumbered by all the other animals. We believe that agency will give the agentic animals such a drastic advantage, that they will seem to take over the world in a very short amount of time.
(Etc etc)
(Possible that this is in one of the things you cite, and either I missed it or I am failing to remember it)

A Ray Mar 5, 2022, 1:50 AM
LW: 3 AF: 2
AF
on: Alex Ray’s Shortform
Hacking the Transformer Prior
Neural Network Priors
I spend a bunch of time thinking about the alignment of the neural network prior for various architectures of neural networks that we expect to see in the future.
Whatever alignment failures are highly likely under the neural network prior are probably worth a lot of research attention.
Separately, it would be good to figure out knobs/levers for changing the prior distribution to be more aligned (or produce more aligned models). This includes producing more interpretable models.
Analogy to Software Development
In general, I am able to code better if I have access to a high quality library of simple utility functions. My goal here is to sketch out how we could do this for neural network learning.
Naturally Occurring Utility Functions
One way to think about the induction circuits found in the Transformer Circuits work is that they are “learned utility functions”. I think this is the sort of thing we might want to provide the networks as part of a “hacked prior”
A Language for Writing Transformer Utility Functions
Thinking Like Transformers provides a programming language, RASP, which is able to express simple functions in terms of how they would be encoded in transformers.
Concrete Research Idea: Hacking the Transformer Prior
Use RASP (or something RASP-like) to write a bunch of utility functions (such as the induction head functions).
Train a language model where a small fraction of the neural network is initialized to your utility functions (and the rest is initialized normally).
Study how the model learns to use the programmed functions. Maybe also study how those functions change (or don’t, if they’re frozen).
Future Vision
I think this could be a way to iteratively build more and more interpretable transformers, in a loop where we:
- Study transformers to see what functions they are implementing
- Manually implement human-understood versions of these functions
- Initialize a new transformer with all of your functions, and train it
- Repeat
If we have a neural network that is eventually entirely made up of human-programmed functions, we probably have an Ontologically Transparent Machine. (AN: I intend to write more thoughts on ontologically transparent machines in the near future)

A Ray Mar 4, 2022, 10:37 PM
LW: 8 AF: 5
AF
on: Musings on the Speed Prior
I think there’s a lot going on with your equivocating the speed prior over circuits w/ a speed prior over programs.
I think a lot of the ideas in this direction are either confused by the difference between circuit priors and program priors, or at least treating them as equivalent. Unfortunately a lot of this is vague until you start specifying the domain of model. I think specifying this more clearly will help communicating about these ideas. To start with this myself, when I talk about circuit induction, I’m talking about things that look like large randomly initialized bayes nets (or deep neural networks).
Program Induction Priors are bad: I would claim that any program induction priors (simplicity prior, speed prior, others) are almost always a bad fit for developing useful intuitions about the behavior of large random bayes net machines.
Confusion between circuit induction speed prior and simplicity prior: I think your point about double descent is wrong — in particular, the speed is largely unchanged in double descent experiments, since the *width* is the parameter being varied, and all deep neural networks of the same depth have approximately the same speed (unless you mean something weird by speed).
Circuit Simplicity: You give circuit-size and circuit-depth as examples of a “speed prior”, which seems pretty nonstandard, especially when describing it as “not the simplicity prior”.
More than Speed and Simplicity: I think there are other metrics that provide interesting priors over circuits, like likelihood under some initialization distribution. In particular, I think “likelihood under the initialization distribution” is the prior that matters most, until we develop techniques that let us “hack the prior”.
Connection to Infinite-Size Neural Networks: I think research about neural networks approaching/at the infinite limit looks a lot like physics about black holes — and similarly can tell us interesting things about dynamics we should expect. In particular, for systems optimized by gradient descent, we end up with infinitesimal/nonexistent feature learning in the limit — which is interesting because all of the sub-modules/sub-circuits we start with are all we’ll ever have! This means that even if there are “simple” or “fast” circuits, if they’re not likely under the initialization distribution, then we expect they’ll have a vanishingly small effect on the output. (One way of thinking about this is in terms of the NTK, that even if we have extremely powerfully predictive modules, their predictive power will be overwhelmed by the much more common and simple features)
Hacking the Prior: Right now we don’t have a good understanding of the behavior of partially-hand coded neural networks, but I think they could serve as a new/distinct class of models (with regards to what functions are likely under the initialization distribution). Concretely, this could look like us “hand-programming” circuits or parts of neural networks, then randomly initializing the rest, and see if during training the model learns to use those programmed functions.

A Ray Mar 1, 2022, 12:35 AM
LW: 7 AF: 3
AF
on: Alex Ray’s Shortform
Interpretability Challenges

Inspired by a friend I’ve been thinking about how to launch/run interpretability competitions, and what the costs/benefits would be.
I like this idea a lot because it cuts directly at one of the hard problems of spinning up in interpretability research as a new person. The field is difficult and the objectives are vaguely defined; it’s easy to accidentally trick yourself into seeing signal in noise, and there’s never certainty that the thing you’re looking for is actually there.
On the other hand, most of the interpretability-like interventions in models (e.g. knowledge edits/updates to transformers) make models worse and not better—they usually introduce some specific and contained deficiency (e.g. predict that the Eiffel Tower is in Rome, Italy).
So the idea for Interpretability Challenges would be to use existing methods (or possibly invent new ones) to inject concrete “things to find” inside of models, release those models as challenges, and then give prizes for finding things.
Some ways this might work:
- Super simple challenge: use editing techniques like ROME to edit a model, upload to google drive, and post a challenge to lesswrong. I’d probably personally put up a couple of prizes for good writeups for solutions.
- CTF (Capture the Flag): the AI Village has been interested in what sorts of AI challenges/competitions could be run in tandem with infosec conferences. I think it would be pretty straightforward to build some interpretability challenges for the next AI Village CTF, or to have a whole interpretability-only CTF by itself. This is exciting to me, because its a way to recruit more people from infosec into getting interested in AI safety (which has been a goal of mine for a while).
- Dixit-rules challenge league: One of the hard problems with challenges like this is how to set the difficulty. Too hard and no one makes progress. Too easy and no one learns/grows from it. I think if there were a bunch of interested people/groups, we could do a dixit style tournament: Every group takes turns proposing a challenge, and gets the most points if exactly one other group solves it (they don’t get points if everyone solves it, or if no one solves it). This has a nice self-balancing force, and would be good if there wanted to be an ongoing group who built new challenges as new interpretability research papers were published.
Please reach out to me if you’re interested in helping with efforts like this.

A Ray Feb 28, 2022, 10:54 PM
14 points
on: Alex Ray’s Shortform
My Cyberwarfare Concerns: A disorganized and incomplete list
- A lot of internet infrastructure (e.g. BGP / routing) basically works because all the big players mostly cooperate. There have been minor incidents and attacks but nothing major so far. It seems likely to be the case that if a major superpower was backed into a corner, it could massively disrupt the internet, which would be bad.
- Cyberwar has a lot of weird asymmetries where the largest attack surfaces are private companies (not militaries/governments). This gets weirder when private companies are multinational. (Is an attack on google an attack on ireland? USA? Neither/both?)
- It’s unclear who is on whose side. The Snowden leaks showed that american intelligence was hacking american companies private fibers on american soil, and the trust still hasn’t recovered. It’s a low-trust environment out there, which seems (to me) to make conflict more likely to start, and harder to contain and extinguish once started.
- There is no good international “law of war” with regards to cyberwarfare. There are some works-in-progress which have been slowly advancing, but there’s nothing like the geneva convention yet. Right now existing cyber conflicts haven’t really pushed the “what is an illegal attack” sense (in the way that land mines are “illegal” in war), and the lack of clear guidance here means that in an all-out conflict there isn’t much in the way of clear limitations.
- Many cyber attacks are intentionally vague or secret in origin. Some of this is because groups are distributed and only loosely connected to national powers (e.g. via funding, blind eyes, etc) and some because it’s practically useful to have plausible deniability. This really gets in the way of any sort of “cease fire” or armistice agreements—if a country comes to the peace treaty table for a given cyber conflict, this might end up implicating them as the source of an attack.
- Expanding the last point more, I’m worried that there are a lot of “ratchet-up” mechanisms for cyberwarfare, but very few “ratchet-down” mechanisms. All of these worries somewhat contribute to a situation where if the currently-low-grade-burning-cyberwar turns into more of an all-out cyberwar, we’ll have very few tools for deescalation.
- Relating this to my concerns about AGI safety, I think an ‘all-out cyberwar’ (or at least a much larger scale one) is one of the primary ways to trigger an AGI weapons development program. Right now it’s not clear to me that much of weapons development budget is spent on cyberweapons (as opposed to other capabilities like SIGINT), but a large-scale cyberwar seems like a reason to invest more. The more money is spent on cyberweapons development, the more likely I think it is that an AGI weapons program is started. I’m not optimistic about the alignment or safety of an AGI weapons program.
Maybe more to come in the future but that’s it for now.

A Ray Feb 18, 2022, 3:59 AM
3 points
in reply to: StellaAthena’s comment on: Compute Trends Across Three eras of Machine Learning
I think that the authors at least did some amount of work to distinguish the eras, but agree more work could be done.
Also I agree w/ Stella here that Turing, GPT-J, GShard, and Switch are probably better fit into the “large scale“ era.

A Ray Feb 17, 2022, 10:34 PM
LW: 2 AF: 2
AF
on: Alex Ray’s Shortform
I with more of the language alignment research folks were looking into how current proposals for aligning transformers end up working on S4 models.
(I am one of said folks so maybe hypocritical to not work on it)
In particular it seems like there’s way in which it would be more interpretable than transformers:
- adjustable timescale stepping (either sub-stepping, or super-stepping time)
- approximately separable state spaces/dynamics—this one is crazy conjecture—it seems like it should be possible to force the state space and dynamics into separate groups, in ways that would allow analysis of them in isolation or in relation to the rest of the model
It does seem like they’re not likely to be competitive with transformers for short-context modeling anytime soon, but if they end up being differentially alignment-friendly, then we could instead try to make them more competitive.
(In general I think it’s much easier to make an approach more competitive than it is to make it more aligned)

A Ray Feb 15, 2022, 7:27 AM
7 points
on: Idea: build alignment dataset for very capable models
I work on this sort of thing at OpenAI.
I think alignment datasets are a very useful part of a portfolio approach to alignment research. Right now I think there are alignment risks/concerns for which datasets like this wouldn’t help, but also there are some that it would help for.
Datasets and benchmarks more broadly are useful for forecasting progress, but this assumes smooth/continuous progress (in general a good assumption—but also good to be wary of cases where this isn’t the case).
Some thoughts from working on generating datasets for research, and using those datasets in research:
- Start by building tiny versions of the dataset yourself
- It’s good to switch early to paying labelers/contractors to generate and labels—they won’t be perfect at first, so there’s a lot of iterating in clarifying instructions/feedback/etc
- It’s best to gather data that you’d want to use for research right away, not for some nebulous possible future research
- Getting clean benchmarks that exhibit some well-defined phenomena is useful for academics and grad students
- When in doubt, BIG-Bench is a good place to submit these sorts of tiny evaluative datasets
- Where possible, experiment with using models to generate more data (e.g. with few-shot or generative modeling on the data you have)
- Sometimes a filter is just as good as data (a classifier that distinguishes data inside the desired distribution)
I think this is a great idea, but would be best to start super small. It sounds right now like a huge project plan, but I think it could be road-mapped into something where almost every step along the path produces some valuable input.
Given the amount of funding available from charitable sources for AI alignment research these days, I think a good thing to consider is figuring out how to make instructions for contractors to generate the data, then getting money to hire the contractors and just oversee/manage them. (As opposed to trying to get volunteers to make all the data)

A Ray Feb 12, 2022, 7:14 PM
4 points
on: [Linkpost] TrojanNet: Embedding Hidden Trojan Horse Models in Neural Networks
I worry a little bit about this == techniques which let you hide circuits in neural networks. These “hiding techniques” are a riposte to techniques based on modularity or clusterability—techniques that explore naturally emergent patterns.^[1] In a world where we use alignment techniques that rely on internal circuitry being naturally modular, trojan horse networks can avoid various alignment techniques.
I expect this to happen by default for a bunch of reasons. An easy one to point to is the “free software” + “crypto anarchist” + “fuck your oversight” + “digital privacy” cluster—which will probably argue that the government shouldn’t infringe your right to build and run neural networks that are not-aligned. Similar to how encrypting personal emails subverts attempts to limit harms of email, “encrypting neural network functions” can subvert these alignment techniques.
To me, it seems like this system ends up like a steganography equilibrium—people find new hiding techniques, and others find new techniques to discover hidden information. As long as humans are both sides of this, I expect there to be progress on both sides. In the human-vs-human version of this, I think it’s not too unevenly matched.
In cases where its human-vs-AI, I strongly expect the AI wins in the limit. This is in part why I’m optimistic about things like ELK solutions which might be better at finding “hidden modules” not just naturally occurring ones.
1. ^
  The more time I spend with these, the more I think the idea of naturally occuring modularity or clusterability makes sense / seems likely, which has been a positive update for me