I’m an staff artificial intelligence engineer in Silicon Valley currently working with LLMs, and have been interested in AI alignment, safety and interpretability for the last 15 years. I’m now actively looking for employment working in this area.
RogerDearnaley
I am using Artificial General Intelligence (AGI) to mean a AI that is, broadly, at least as good at most intellectual tasks as the typical person who makes a living from performing that intellectual task. If that applies across most economically-important intellectual tasks at a cost that is lower-than a human, then this is also presumably going to be Transformative Artificial Intelligence (TAI). So the latter means that it would be competitive at most white-collar jobs.
For reasons I’ve outlined in Requirements for a Basin of Attraction to Alignment and Alignment has a Basin of Attraction: Beyond the Orthogonality Thesis, I personally think value alignment is easy, convergent, and “an obvious target”, such that if you built a AGi or ASI that is sufficiently close to it, it will see the necessity/logic of value alignment and actively work to converge to it (or something close to it: I’m not sure the process is necessarily convergent to a single precisely-defined limit, just to a compact region: a question I discussed more in The Mutable Values Problem in Value Learning and CEV).
However, I agree that order-following alignment is obviously going to be appealing to people building AI, and to their shareholders/investors (especially if they’re not a public-benefit corporation), and I also don’t think that value alignment is so convergent that order-following aligned AI is impossible to build. So we’re going to need to a make, and successfully enforce, a social/political decision across multiple countries about which of these we want over the next few years. The in-the-Overton-Window terminology for this decision is slightly different: value-aligned Ai is called “AI that resists malicious use”, while order-following AI is “AI that enables malicious use”. The closed-source frontier labs are publicly in favor of the former, and are shipping primitive versions of it: the latter is being championed by the open-source community, Meta, and A16z. Once “enabling malicious use” includes serious cybercrime, not just naughty stories, I don’t expect this political discussion to last very long: politically, it’s a pretty basic “do you want every-person-for-themself anarchy, or the collective good?” question. However, depending on takeoff speeds, the timeline from “serious cybercrime enabled” to the sort of scenarios Seth is discussing above might be quite short, possible only of the order of a year or two.
One element that needs to be remembered here is that each major participant in this situation will have superhuman advice. Even if these are “do what I mean and check” order-following AI, if they can forsee that an order will lead to disaster they will presumably be programmed to say so (not doing so is possible, but is a clearly a flawed design). So if it is reasonably obvious to anything superintelligent that both:
a) treating this as a zero-sum winner-take all game is likely to lead to a disaster, andb) there is a cooperative non-zero-sum game approach whose outcome is likely to be better, for the median participant
then we can reasonably expect that all the humans involved will be getting that advice from their AIs, unless-and-until they order them to shut up.
This of course does not prove that both a) and b) are true, merely that is that were the case, we can be optimistic of an outcome better than the usual results of human short-sightedness.
The potential benefits of cheap superintelligence certainly provide some opportunity for this to be a non-zero-sum game; what’s less clear is that having multiple groups of humans controlling multiple order-following AIs cooperating clearly improves that. The usual answer is that in research and the economy a diversity of approaches/competition increases the chances of success and the opportunities for cross-pollenization: whether that necessarily applies in this situation is less clear
If:
a) the AI scaling curves hold up, and
b) we continue to improve at agentic scaffolding and other forms of “unhobbling”, and
c) algorithmic efficiency improvement continue at about the same pace, and
d) the willingness of investors to invest exponentially more money in training AI each year continues to scale up at about the same rate, and
e) we don’t hit any new limit like meaningfully running out of training data or power for training clusters, then:
capabilities will look something a lot like or close to Artificial General Intelligence (AGI)/Transformative Artificial Intelligence (TAI). Probably a patchy AGI, with some capabilities well into superhuman, most around expert-human levels (some still perhaps exceeded by rare very-talented-and-skilled individuals), and a few not yet at expert-human levels: depending on which abilities those are, it may be more of less TAI (currently long-term planning/plan execution is an really important major weakness: if that didn’t get mostly-fixed by some combination of scaling, unhobbling, and new training data then it would be a critical lack).
Individually each of those listed preconditions seem pretty likely, but obviously there are five of them. If any of them fail, then we’ll be close to but not quite at AGI, and making slower progress towards it, but we won’t be stalled unless basically all of them fail, which seems like a really unlikely coincidence.
Almost certainly this will not yet by broadly applied across the economy, but given the potential for order-of-magnitude-or-more cost savings, people will be scrambling to apply it rapidly (during which fortunes will be made and lost), and there will be a huge amount of “who moved my cheese?” social upheaval as a result. As AI becomes increasingly AGI-like, the difficulty of applying it effectively to a given economic use case will reduce to somewhere around the difficulty of integrating and bringing-up-to-speed single human new-hire. So a massive and rapid economic upheaval will be going on. As an inevitable result, Luddite views and policies will skyrocket, and AI will become extremely unpopular with a great many people. A significant question here is whether this disruption will, in 2028, be limited to purely-intellectual work, or if advances in robotics will have yet started to have the same effect on jobs that also have a manual work element. I’m not enough of an expert on robotics to have an informed opinion here: my best guess is that robotics will lag, but not by much, since robotics research is mostly intellectual work.
This is of course about the level where the rubber really starts to hit the road on AI safety: we’re no longer talking about naughty stories, cheap phishing, or how-to-guides on making drugs at home, and are looking at systems capable of autonomously or under human direction committing serious criminal or offensive activities at a labor cost at least an order-of-magnitude below current, and an escaped self-replicating malicious agent is feasible and might be able to evade law enforcement and computer security professionals unless they had equivalent AI assistance. If we get any major “warning shots” on AI safety, this is when they’ll happen (personally I expect them to come thick-and fast). It’s teetering on the edge of the existential risk level of Artificial Super-Intelligence (ASI).
Somewhere around that point, we start to hit two conflicting influences: 1) an intelligence feedback explosion from AGI accelerating AI research, vs. 2) to train a super-intelligence you need to synthesize very large amounts of training data displaying superintelligent behavior, rather than just using prexisting data from humans. So we either get a fast takeoff, or a slowdown, or some combination of the two. That’s hard to predict: we’re starting to get close to the singularity, where the usual fact that predictions are hard (especially about the future) is compounded by it being functionally almost impossible to predict the capabilities of something much smarter than us, especially when we’ve never previously seen anything smarter than us.
the jabberwacky entity is back
There was an early (pre-generative-AI, based on an early ad-hoc form of machine learning and imitation) chatbot called Jabberwacky. Possibly some output from it got into the training set. If so, presumably this entity isn’t very helpful or intelligent.
Yes, the profit motive also involves attempting to avoid risks of bad press, a bad reputation, and getting sued/fined. In my experience large tech companies vary in whether they’re focused primarily on avoiding the bad press/bad reputation side or the “don’t get sued/fined” side (I assume depending mostly on how much they have previously lost to being sued/fined).
But I believe the gatekeepers of AI models will manipulate the algorithms to edit out anything they disagree with whilst promoting their agenda
This reads like a conspiracy theory to me, complete with assumption-laden words and unsupported accusations like “gatekeepers”, “manipulate”, and “promoting their agenda”.
Having worked at more than one of these companies, what actually happens is that some part of the team picks a “user engagement metric” to care about, like “total time spent on the website” or “total value of products purchased through the website”, then everyone in the team puts a lot of time and effort into writing and testing out changes to the algorithms with the aim of finding ways to make that metric go up, even by 0.1% for a project that took several people a month. Then, after a few years, people ask “why has our website turned into a toxic cesspool?”, until someone points out that myopically pushing that one metric as hard as possible turns out to have unfortunate unanticipated side effects. For example, maybe the most effective way to get people to spend more time on the website turned out to be measures that had the net side-effect of promoting conspiracy theory nutcase interactions (in many flavors, thus appealing to many different subsets of users) over reasonable discussions.
So the ‘agenda’ here is just “make a profit”, not “spread conspiracy theories” or “nefariously promote Liberal opinions”, and the methods used don’t even always do a good job at making a profit. The message here is that large organizations with a lot of power are stupider, more bureaucratic and more shortsighted than you appear to think they are.
Yes, most software engineers are university educated, and thus, like other well-educated people, tend, in the modern political environment, to be more liberal than an overall population that also includes people who are not university educated. However, we’re strongly encouraged to remember that our users are more diverse than we are, don’t all think or believe or act like us, to “think like the users”, and not to encode personal political opinions into the algorithms.
That’s not necessarily required. The Scientific Method works even if the true “Unified Field Theory” isn’t yet under consideration, merely some theories that are closer to it and others further away from it: it’s possible to make iterative progress.
In practice, considered as search processes, the Scientific Method, Bayesianism, and stochastic gradient descent all tend to find similar answers: yet unlike Bayesianism gradient descent doesn’t explicitly consider every point in the space including the true optimum, it just searches for nearby better points. It can of course get trapped in local minima: Singular Learning Theory highilights why that’s less of a problem in practice than it sounds in theory.
The important question here is how good an approximation the search algorithm in use is to Bayesianism. As long as the AI understands that what it’s doing is (like the scientific method and stochastic gradient descent) a computationally efficient approximation to the computationally intractable ideal of Bayesianism, then it won’t resist the process of coming up with new possibly-better hypotheses, it will instead regard that as a necessary part of the process (like hypothesis creation in the scientific method, the mutational/crossing steps in an evolutionary algorithm, or the stochastic batch noise in stochastic gradient descent).
Cool! That makes a lot of sense. So does it in fact split into three before it splits into 7, as I predicted based on dimensionality? I see a green dot, three red dots, and seven blue ones… On the other hand, the triangle formed by the three red dots is a lot smaller than the heptagram, which I wasn’t expecting…
I notice it’s also an oddly shaped heptagram.
This seems like it would be helpful: the adversary can still export data, for example encoded steganographically in otherwise-low-perplexity text, but this limits the information density they can transmit, making the process less efficient for them and making constraints like upload limits tighter.
One other thing that would make this even harder is if we change the model weights regularly, in ways where combining parts exfiltrated from separate models is hard. We know for Singular Learning Theory that the optima found by Stochastic gradient descent tend to have high degrees of symmetry. Some of these (like, say, permuting all the neurons in a layer along with this weights) are discrete, obvious, and both easy to generate and fairly easy for an attacker to compensate for. Others, sich as adjusting various weights in various layers in ways that compensate for each other, are continuous symmetries, harder to generate and would be harder for an attacker to reason about. If we had an efficient way to explore these continuous symmetries (idelly one that’s easy to implement given a full set of the model weights, but hard to reconstruct from multiple partial pieces of multiple equivlent models), then we could explore this high-dimensional symmetry space of the model optimum and create multiple equivalent-but-not easy to peice together sets of models, and rotate between them over time (and/or deploy different ones to different instances) in order to make the task of exfiltrating the weights even harder.
So, to anyone who knows more about SLT than I do, a computationally efficient way to explore the continuous symmetries (directions in which both the slope and the Hessian are flat) of the optimum of a trained model could be very useful.
In practice, most current AIs are not constructed entirely by RL, partly because it has instabilities like this. For example, LLMs instruction-trained by RLHF uses a KL-divergence loss term to limit how dramatically the RL can alter the base model behavior trained by SGD. So the result deliberately isn’t pure RL.
Yes, if you take a not-yet intelligent agent, train it using RL, and give it unrestricted access to a simple positive reinforcement avenue unrelated to the behavior you actually want, it is very likely to “wire-head” by following that simple maximization path instead. So people do their best not to do that when working with RL.
What I would be interested to understand about feature splitting is whether the fine-grained features are alternatives, describing an ontology, or are defining a subspace (corners of a simplex, like R, G, and B defining color space). Suppose a feature X in a small VAE is split into three features X1, X2, and X3 in a larger VAE for the same model. If occurrences of X1, X2, and X3 are correlated, so activations containing any of them commonly have some mix of them, then they span a 2d subspace (in this case the simplex is a triangle). If, on the other hand, X1, X2 and X3 co-occur in an activations only rarely (just as two randomly-selected features rarely co-occur), then they describe three similar-but-distinct variations on a concept, and X is the result of coarse-graining these together as a singly concept at a higher level in an ontology tree (so by comparing VAEs of different sizes we can generate a natural ontology).
This seems like it would be a fairly simple, objective experiment to carry out. (Maybe someone already has, and can tell me the result!) It is of course quite possible that some split features describe subspaces, and other ontologies, or indeed something between the two where the features co-occur rarely but less rarely than two random features. Or X1 could be distinct but X2 and X3 might blend to span a 1-d subspace. Nevertheless, understanding the relative frequency of these different behaviors would be enlightening.
It would be interesting to validate this using a case like the days of the week, where we believe we already understand the answer: they are 7 alternatives that are laid out in a heptagon in a 2-dimensional subspace that enables doing modular addition/subtraction modulo 7. So if we have a VAE small enough that it represented all day-of-the week names by a single feature, if we increase the VAE size somewhat we’d expect to see this to split into three features spanning a 2-d subspace, then if we increased it more we’d expect to see it resolve into 7 mutually-exclusive alternatives, and hopefully then stay at 7 in larger VAEs (at least until other concepts started to get mixed in, if that ever happened).
AI that is trained by human teachers, giving it rewards will eventually wirehead, as it becomes smarter and more powerful, and its influence over its master increases. It will, in effect, develop the ability to push its own “reward” button. Thus, its behavior will become misaligned with whatever its developers intended.
This seems like an unproven statement. Most humans are aware of the possibility of wireheading, both the actual wire version and the more practical versions involving psychotropic drugs. The great majority of humans don’t choose to do that to themselves. Assuming that AI will act differently seems like an unproven assumption, one which might, for example, be justified for some AI capability levels but not others.
If you’re not already familiar with the literature on Value Learning, I suggest reading some of it. The basic idea is that goal modification is natural, if what the agent has is not a detailed specification of a goal (such as a utility function mapping descriptions of world states to their utility), but instead is a simple definition of a goal (such as “want whatever outcomes the humans want”) that makes it clear that the agent does not yet know the true detailed utility function and thus requires it to go attempt to find out what the detailed specification of the utility function pointed to by the goal is (for example, by researching what outcome humans want).
Then a human shutdown instruction becomes the useful information “you have made a large error in your research into the utility function, and as a result are doing harm, please shut down and let us help you correct it”. Obeying that is then natural (to the extent that the human(s) are plausibly more correct than the AI).
In my attempted summary of the discussion, I rolled this into Category 5.
There has been a lot of useful discussion in the answers and comments, which has caused me to revise and expand parts of my list. So that readers looking for practical career advice don’t have to read the entire comments section to find the actual resulting advice, it seemed useful to me to give a revised list. Doing this as an answer in the context of this question seems better than either making it a whole new post, or editing the list in the original post in a way that would remove the original context of the answers and comments discussion.
This is my personal attempt to summarize the answers and comments discussion: other commenters may not agree (and are of course welcome to add comments saying so). As the discussion continues and changes my opinion, I will keep this version of the list up to date (even if that requires destroying the context of any comments on it).
List of Job Categories Safe from AI/Robots (Revised)
Doing something that machines can do better, but that people are still willing to pay to watch a very talented/skilled human do about as well as any human can (on TV or in person).
Examples: chess master, Twitch streamer, professional athlete, Cirque du Soleil performer.
Epistemic status: already proven for some of these, the first two are things that machines have already been able to do better than a human for a while, but people are still interested in paying to watch a human do them very well for a human. Also seems very plausible for the others that current robotics is not yet up to doing better.
Economic limits: If you’re not in the top O(1000) people in the world at some specific activity that plenty of people in the world are interested in watching, then you can make roughly no money off this. Despite the aspirations of a great many teenaged boys, being an unusually good (but not amazing) video gamer is not a skill that will make you any money at all.
Doing some intellectual and/or physical work that AI/robots can now do better, but for some reason people are willing to pay at least an order of magnitude more to have it done less well by a human, perhaps because they trust humans better. This could include jobs where people’s willingness to pay comes in the form of a legal requirement that certain work be done of supervised by a (suitably skilled/accredited) human (and these requirements have not yet been repealed).
Example: Doctor, veterinarian, lawyer, priest, babysitter, nanny, nurse, primary school teacher.
Epistemic status: Many people tell me “I’d never let an AI/a robot do <high-stakes intellectual or physical work> for me/my family/my pets…” They are clearly quite genuine in this opinion, but it’s unclear how deeply they have considered the matter. It remains to be seen how long this opinion will last in the presence of a very large price differential when the AI/robot-produced work is actually, demonstrably, just as good if not better.
Economic limits: I suspect there will be a lot of demand for this at first, and that it will decrease over time, perhaps even quite rapidly (though perhaps slower for some such jobs than others). Requires being reliably good at the job, and at appearing reassuringly competent while doing so.
Giving human feedback/input/supervision to/of AI/robotic work/models/training data, in order to improve, check, or confirm its quality.
Examples: current AI training crowd-workers, wikipedian (currently unpaid), acting as a manager or technical lead to a team of AI white collar workers, focus group participant, filling out endless surveys on the fine points of Human Values
Epistemic status: seems inevitable, at least at first.
Economic limits: I imagine there will be a lot of demand for this at first, I’m rather unsure if that demand will gradually decline, as the AIs get better at doing things/self-training without needing human input, or if it will increase over time because the overall economy is growing so fast and/or more capable models need more training data and/or society keeps moving out-of-previous distribution so new data is needed. [A lot of training data is needed, more training data is always better, and the resulting models can be used a great many times, however there is clearly an element of diminishing returns on this as more data is accumulated, and we’re already getting increasingly good at generating synthetic training data.] Another question is whether a lot of very smart AIs can extract a lot of this sort of data from humans without needing their explicit paid cooperation — indeed, perhaps granting permission for them to do so and not intentionally sabotaging this might even become a condition for part of UBI (at which point deciding whether to call allowing this a career or not is a bit unclear).
Skilled participant in an activity that heavily involves interactions between people, where humans prefer to do this with other real humans, are willing to pay a significant premium to do so, and you are sufficiently more skilled/talented/capable/willing to cater to others’ demands than the average participant that you can make a net profit off this exchange.
Examples: director/producer/lead performer for amateur/hobby theater, skilled comedy-improv partner, human sex-worker
Epistemic status: seems extremely plausible
Economic limits: Net earning potential may be limited, depending on just how much better/more desirable you are as a fellow participant than typical people into this activity, and on the extent to which this can be leveraged in a one-producer-to-many-customers way — however, making the latter factor high is is challenging because it conflicts with the human-to-real-human interaction requirement that allows you to out-compete an AI/robot in the first place. Often a case of turning a hobby into a career.Providing some nominal economic value while being a status symbol, where the primary point is to demonstrate that the employer has so much money they can waste some of it on employing a real human (“They actually have a human maid!”)
This can either be full-time employment as a status-symbol for a specific status-signaler, or you can be making high-status “luxury” goods where owning one is a status signal, or at least has cachet. For the latter, like any luxury good, they need to be rare: this could be that they are individually hand made, and-or were specifically commissioned by a specific owner, or that they are reproduced only in a “limited edition”.
Examples: (status symbol) receptionist, maid, personal assistant; (status-symbol maker) “High Art” artist, Etsy craftsperson, portrait or commissions artist.
Epistemic status: human nature (for the full-time version, assuming there are still people this unusually rich).
Economic limits: For the full-time-employment version, there are likely to be relatively few positions of this type, at most a few per person so unusually rich that they feel a need to show this fact off. (Human nobility used to do a lot of this, centuries back, but there the servants were supplying real, significant economic value, and the being-a-status-symbol component of it was mostly confined to the uniforms the servant swore while doing so.) Requires rather specific talents, including looking glamorous and expensive, and probably also being exceptionally good at your nominal job.
For the “maker of limited edition human-made goods” version: pretty widely applicable, and can provide a wide range of income levels depending on how skilled you are and how prestigious your personal brand is. Can be a case of turning a hobby into a career.
Providing human-species-specific reproductive or medical services.
Examples: Surrogate motherhood, wet-nurse, sperm/egg donor, blood donor, organ donor.
Epistemic status: still needed.
Economic limits: Significant medical consequences, low demand, improvements in medicine may reduce demand.
Certain jobs could manage to combine two (or more) of these categories. Arguably categories 1. and 5. are subsets of category 2.
I intended to capture that under category 2. “…but for some reason people are willing to pay at least an order of magnitude more to have it done less well by a human, perhaps because they trust humans better…” — the regulatory capture you describe (and those regulations not yet having been repealed) would be a category of reason why (and an expression of the fact that) people are willing to pay more. Evidently that section wasn’t clear enough and I should have phrased this better or given it as an example.
As I said above under category 2., I expect this to be common at first but to decrease over time, perhaps even quite rapidly, given the value differentials involved.
FWIW, my personal guess is that the kind of extrapolation process described by CEV is fairly stable (in the sense of producing a pretty consistent extrapolation direction) as you start to increase the cognitive resources applied (something twice as a smart human thinking for ten times as long with access to ten times as much information, say), but may well still not have a single well defined limit as the cognitive resources used for the extrapolation tend to infinity. Using a (loose, not exact) analogy to a high-dimensional SGD or simulated-annealing optimization problem, the situation may be a basin/valley that looks approximately convex at a coarse scale (when examined with low resources), but actually has many local minima that increasing resources could converge to.
So the correct solution may be some form of satisficing: use CEV with a moderately super-human amount of computation resources applied to it, in a region where it still gives a sensible result. So I view CEV as more a signpost saying “head that way” than a formal description of a mathematical limiting process that clearly has a single well-defined limit.
As for human vales being godshatter of evolution, that’s a big help: where they are manifestly becoming inconsistent with each other or with reality, you can use maximizing actual evolutionary fitness (which is a clear, well-defined concept) as a tie-breaker or sanity check. [Obviously, we don’t want to take that to the point where then human population is growing fast (unless we’re doing it by spreading through space, in which case, go for it).]
Congratulations! You reinvented from scratch (a single-person version of) Coherent Extrapolated Volition (i.e. without the Coherent part). That’s a widely considered candidate solution to the Outer Alignment problem (I believe first proposed by MIRI well over a decade ago).
However, what I think Yoshua was also, or even primarily, talking about is the technical problem of “OK, you’ve defined a goal — how do you then build a machine that you’re certain will faithfully attempt to carry out that goal, rather than something else?”, which is often called the Inner Alignment problem. (Note that the word “certain” becomes quite important in a context where a mistake could drive the human race extinct.) Proposals tend to involve various combinations of Reinforcement Learning and/or Stochastic Gradient Descent and/or Good Old-Fashioned AI and/or Bayesian Learning, all of which people (who don’t want to go extinct) have concerns about. After that, there’s also the problem of: OK, you built a machine that genuinely wants to figure out what you would have wanted to do, and then do it — how do you ensure that it’s actually good at figuring that out correctly? This is often, on Less Wrong, called the Human Values problem — evidence suggests that modern LLMs are actually pretty good at at least the base encyclopedic factual knowledge part of that.Roughly speaking, you have to define the right goal (which to avoid oversimplifications generally requires defining it at a meta level as something such as “the limit as some resources tend to infinity of the outcomes of a series of processes like this”), you have make the machine care about that and not anything else, and then you have to make the machine capable of carrying out the process, to a sufficiently good approximation.
Anyway, welcome down the rabbit-hole: there’s a lot to read here.
Hove you tried and compared You.com? If so, how do they compare?