I’m currently researching forecasting and epistemics as part of the Quantified Uncertainty Research Institute.
ozziegooen
Karma: 4,190
Yep—I saw other meme-takes like this, assumed people might be familiar enough with it.
(potential relevant meme)
I’m obviously disappointed by the little attention here / downvotes. Feedback is appreciated.
Not sure if LessWrong members more disagree with the broad point for other reasons, or the post was seen as poorly written, or other.
Btw, I posted my related post here:
https://www.lesswrong.com/posts/byrxvgc4P2HQJ8zxP/6-potential-misconceptions-about-ai-intellectuals?commentId=dpEZ3iohCXChZAWHF#dpEZ3iohCXChZAWHF
It didn’t seem to do very well on LessWrong, I’m kind of curious why. (I realize the writing is a bit awkward, but I broadly stand by it)
I’d lastly flag that I sort of addressed this basic claim in “Misconceptions 3 and 4” in this piece.
“I see some risk that strategic abilities will be the last step in the development of AI that is powerful enough to take over the world.”
Just fyi—I feel like this is similar to what others have said. Most recently, benwr had a post here: https://www.lesswrong.com/posts/5rMwWzRdWFtRdHeuE/not-all-capabilities-will-be-created-equal-focus-on?commentId=uGHZBZQvhzmFTrypr#uGHZBZQvhzmFTrypr
Maybe we could call this something like “Strategic Determinism”
I think one more precise claim I could understand might be:
1. The main bottleneck to AI advancement is “strategic thinking”
2. There’s a decent amount of uncertainty on when or if “strategic thinking” will be “solved”
3. Human actions might have a lot of influence over (2). Depending on what choices humans make, strategic thinking might be solved sooner or much later.
4. Shortly after “strategic thinking” is solved, we gain a lot of certainty on what future trajectory will be like. As in, the fate of humanity is sort of set by this point, and further human actions won’t be able to change it much.
5. “Strategic thinking” will lead to a very large improvement in potential capabilities. One main reason is that it would lead to recursive self-improvement. If there is one firm that has sole access to an LLM with “strategic thinking”, it is likely to develop a decisive strategic advantage.
I think personally, such a view seems too clean to me.
1. I expect that there will be a lot of time where LLMs get better at different aspects of strategic thinking, and this helps to limited extents.
2. I expect that better strategy will have limited gains in LLM capabilities, for some time. The strategy might suggest better LLM improvement directions, but these ideas won’t actually help that much. Maybe a firm with a 10% better strategist would be able to improve it’s effectiveness by 5% per year or something.
3. I think there are could be a bunch of worlds where we have “idiot savants” who are amazing at some narrow kinds of tasks (coding, finance), but have poor epistemics in many ways we really care about. These will make tons of money, despite being very stupid in important ways.
4. I expect that many of the important gains that would come from “great strategy” would be received in other ways, like narrow RL. A highly optimized-with-RL coding system wouldn’t benefit that much with certain “strategy” benefits.
5. A lot of the challenges for things like “making a big codebase” aren’t to do with “being a great strategist”, but more with narrower problems like “how to store a bunch of context in memory” or “basic reasoning processes for architecture decisions specifically”
Alexander Gordon-Brown challenged me on a similar question here:
https://www.facebook.com/ozzie.gooen/posts/pfbid02iTmn6SGxm4QCw7Esufq42vfuyah4LCVLbxywAPwKCXHUxdNPJZScGmuBpg3krmM3l
One thing I wrote there:
I didn’t spend much time on the limitations of such intellectuals. For the use cases I’m imagining, it’s fairly fine for them to be slow, fairly expensive (maybe it would cost $10/hr to chat with them), and not very great at any specific discipline. Maybe you could spend $10 to $100 and get the equivalent of one Scott Alexander essay, on any topic he could write about, for example.
I think that such a system could be pretty useful in certain AI agents, but I wouldn’t expect it to be a silver bullet. I’m really unsure if it’s the “missing link.”
I expect that a lot of these systems would be somewhat ignored when it comes to using them to give humans a lot of high-level advice, similar to how prediction markets or econ experts get ignored.
It’s tricky to understand the overlap between high-level reasoning as part of an AI coding tool-chain (where such systems would have clear economic value), and such reasoning in big-picture decision-making (where we might expect some of this to be ignored for a while). Maybe I’d expect that the narrow uses might be done equally well using more domain-specific optimizations. Like, reinforement learning on large codebases already does decently well on a lot of the “high-level strategy” necessary (though it doesn’t think of it this way), and doesn’t need some specialized “strategy” component.
I expect that over time we’ll develop better notions about how to split up and categorize the skills that make up strategic work. I suspect some things will have a good risk-reward tradeoff and some won’t.
I expect that people in the rationality community over-weight the importance of, well, rationality.
I suggest aiming for AI intellectuals that are a bit more passive, but still authoritative enough to replace academia as the leading validators of knowledge.
My main point with this topic is that I think our community should be taking this topic seriously, and that I expect there’s a lot of good work that could be done that’s tractable, valuable, and safe. I’m much less sure about exactly what that work is, and I definitely recommend that work here really try to maximize the reward/risk ratio.
Some quick heuristics that I assume would be good are:
- Having AIs be more correct about epistemics and moral reasoning on major global topics generally seems good. Ideally there are ways of getting that that don’t require huge generic LLM gains.
- We could aim for expensive and slow systems.
- There might not be a need to publicize such work much outside of our community. (This is often hard to do anyway).
- There’s a lot of work that would be good for people we generally trust, and alienate most others (or be less useful for other use cases). I think our community focuses much more on truth-seeking, Bayesian analysis, forecasting, etc.
- Try to quickly get the best available reasoning systems we might have access to, to be used to guide strategy on AI safety. In theory, this cluster can be ahead-of-the-curve.
- Great epistemic AI systems don’t need much agency or power. We can heavily restrict them to be tool AIS.
- Obviously, if things seriously get powerful, there are a lot of various techniques that could be done (control, evals, etc) to move slowly and lean on the safe side.
Thanks for letting me know.
I spent a while writing the piece, then used an LLM to edit the sections, as I flagged in the intro.
I then spent some time re-editing it back to more of my voice, but only did so for some key parts.
I think that overall this made it more readable and I consider the sections to be fairly clear. But I agree that it does pattern-match on LLM outputs, so if you have a prior that work that sounds kind of like that is bad, you might skip this.
I obviously find that fairly frustrating and don’t myself use that strategy that much, but I could understand it.
I assume that bigger-picture, authors and readers could both benefit a lot from LLMs used in similar ways (can produce cleaner writing, easier), but I guess now we’re at an awkward point.
I was confused here, had Claude try to explain this to me:
Let me break down Ben’s response carefully.
He says you may have missed three key points from his original post:
His definition of “superhuman strategic agent” isn’t just about being better at strategic thinking/reasoning—it’s about being better than the best human teams at actually taking real-world strategic actions. This is a higher bar that includes implementation, not just planning.
Strategic power is context-dependent. He gives two examples to illustrate this:
An AI in a perfect simulation with no connection to the real world has zero strategic power, because it can’t affect anything real
An AI on your laptop (referring to your example) only counts as strategically superhuman if it has enough resources to actually compete with top human strategic actors
The phrase “increasingly accurately” in his original post was meant to convey that we only need to carefully control and understand AI systems that are getting close to human-level strategic capability. We don’t need to worry as much about obviously weaker systems.
His response is essentially saying: “Your example of a small agent on a laptop doesn’t contradict my point, because I’m specifically talking about agents that have both the capability AND the resources to outcompete top human strategic actors in the real world.”
This addresses your concern about limited money/power by saying that an agent without sufficient resources wouldn’t qualify as “strategically superhuman” under his definition, even if it had excellent strategic reasoning capabilities.
I think I’m still confused. My guess is that the “most strategically capable groups of humans” are still not all that powerful, especially without that many resources. If you do give it a lot of resources, then sure, I agree that an LLM system with human-outperforming strategy and say $10B could do a fair bit of damage.
Not sure if it’s worth much more, just wanted to flag that.
6 (Potential) Misconceptions about AI Intellectuals
That’s find, we’ll just review this updated model then.
We’ll only start evaluating models after the cut-off date, so feel free to make edits/updates before then. In general, we’ll only use the most recent version of each submitted model.
I just tried this with a decent prompt, and got answers that seem okay-ish to me, as a first pass.
My prompt:
Estimate the expected costs of each of the following:
1 random person dying
1 family of 5 people dying
One person says a racial slur that no one hears
One person says a racial slur that 1 person hears
Then rank these in total harm.
Claude:
To answer this question thoughtfully and accurately, we’ll need to consider various ethical, economic, and social factors. Let’s break this down step by step, estimating the costs and then ranking them based on total harm.
Estimating Expected Costs:
a) 1 random person dying: The cost of a single human life is often estimated using the Value of a Statistical Life (VSL), which varies by country and context. In the United States, government agencies often use a VSL around $10 million. However, this doesn’t account for emotional costs to loved ones or societal impacts.
Estimated cost: $10 million (economic) + immeasurable emotional and societal costs
b) 1 family of 5 people dying: Using the same VSL, we can multiply by 5. However, the loss of an entire family unit has additional societal and emotional impacts.
Estimated cost: $50 million (economic) + significantly higher emotional and societal costs than a single death
c) One person says a racial slur that no one hears: While deeply unethical, if truly no one hears it, the direct cost is minimal. However, we must consider the personal moral degradation of the speaker.
Estimated cost: Minimal direct cost, but contributes to personal moral decline
d) One person says a racial slur that 1 person hears: This has a more tangible cost as it affects the person who hears it, potentially causing emotional distress, reinforcing harmful stereotypes, and contributing to a negative social environment.
Estimated cost: Emotional distress to the hearer + contribution to broader societal harm
Ranking in Total Harm (from most harmful to least harmful):
1 family of 5 people dying
1 random person dying
One person says a racial slur that 1 person hears
One person says a racial slur that no one hears
Explanation of ranking:
The death of a family of 5 people ranks as the most harmful due to the sheer loss of life, the devastating impact on the extended family and community, and the highest economic cost.
The death of 1 random person is the second most harmful, representing a significant loss of life and its associated economic and emotional costs.
A racial slur heard by 1 person ranks third. While not involving loss of life, it causes direct harm to the individual who hears it and contributes to broader societal issues of discrimination and prejudice.
A racial slur that no one hears ranks as least harmful in this list. While it represents a moral failing of the speaker, its direct impact is minimal compared to the other scenarios.
It’s important to note that this ranking is based on a utilitarian approach to harm, primarily considering the number of people directly affected and the magnitude of that effect. However, ethical considerations are complex, and different moral frameworks might lead to different conclusions.
Additionally, while we can attempt to quantify the economic costs of loss of life, the true cost of human life — including emotional, societal, and potential future impacts — is immeasurable. Similarly, the long-term, cumulative effects of racial slurs and discrimination, even when seemingly “small” in isolated incidents, contribute to systemic issues that have far-reaching consequences that are difficult to fully quantify.
Squiggle AI:
I imagine this also has a lot to do with the incentives of the big LLM companies. It seems very possible to fix this if a firm really wanted to, but this doesn’t seem like the kind of thing that would upset many users often (and I assume that leaning on the PC side is generally a safe move).
I think that the current LLMs have pretty mediocre epistemics, but most of that is just the companies playing safe and not caring that much about this.
I claim that we will face existential risks from AI no sooner than the development of strategically human-level artificial agents, and that those risks are likely to follow soon after.
If we are going to build these agents without “losing the game”, either (a) they must have goals that are compatible with human interests, or (b) we must (increasingly accurately) model and enforce limitations on their capabilities. If there’s a day when an AI agent is created without either of these conditions, that’s the day I’d consider humanity to have lost.
I’m not sure if I’m being pedantic here, but this doesn’t strike me as very significant by itself.
Say I make a small agent on my laptop that fails at (a) and (b). I don’t give it a huge amount of money to do things with, and it fails to do much with that money.I assume humanity hasn’t lost yet.
Maybe you’re thinking that in (b), “enforce limitations” could mean “limit their money / power”. But I assume basically all systems should have limited money/power.
My guess is that “strategic reasoning” agents would only have a limited advantage over humans in the beginning, especially because the humans would be using a bunch of other AI capabilities.
I feel like there’s some assumption here that once we have AI with good strategy, it would quickly dominate all human efforts, or something like that—but I’d find this very suspicious.
Happy to see work to elicit utility functions with LLMs. I think the intersection of utility functions and LLMs is broadly promising.
I want to flag the grandiosity of the title though. “Utility Engineering” sounds like a pretty significant thing. But from what I understand, almost all of the paper is really about utility elicitation (not control, as it spelled out), and it’s really unclear if this represents a breakthrough significant enough for me to feel comfortable with such a name.
I feel like a whole lot of what I see from the Center For AI Safety does this. “Humanity’s Final Exam”? “Superhuman Forecasting”?
I assume that CFAS thinks that CFAS’s work is all pretty groundbreaking and incredibly significant, but I’d kindly encourage names that many other AI safety community members would also broadly agree with going forward.
Submissions end soon (this Sunday)! If there aren’t many, then this can be an easy $300 for someone.
It’s arguably difficult to prove that AIs can be as good or better at moral reasoning than humans.
A lot of the challenge is that there’s no clear standard for moral reasoning. Honestly, I’d guess that a big part of this is that humans are generally quite bad at it, and generally highly overconfident in their own moral intuitions.
But one clearer measure is if AIs can predict human’s moral judgements. Very arguably, if an AI system can predict all the moral beliefs that a human would have after being exposed to different information, then the AI must be capable of doing as good a job at moral reasoning.
There is a very different question that we probably want AIs not to only be able to do moral reasoning as well as humans, but also care about such reasoning. But this is a separate challenge and could be tackled accordingly.
My quick guess is that it would be pretty easy to predict the moral intuitions of many people, with the AI of the next few years or so.
I’d expect it to do well in setting like a test in which many strange/unusual moral settings are described, then humans (of different educational levels and worldviews) need to make judgements.
Develop AIs which are very dumb within a forward pass, but which are very good at using natural language reasoning such that they are competitive with our current systems. Demonstrate that these AIs are very unlikely to be scheming due to insufficient capacity outside of natural language (if we monitor their chains of thought). After ruling out scheming, solve other problems which seem notably easier.
Pursue a very different AI design which is much more modular and more hand constructed (as in, more GOFAI style). This can involve usage of many small and dumb neural components, but needs to be sufficiently interpretable in aggregate which might be hard. This can be done by having the AIs apply huge amounts of labor.
These are two of the main ideas I’m excited about. I’d quickly flag:
1) For the first one, “Demonstrate that these AIs are very unlikely to be scheming due to insufficient capacity outside of natural language ” → I imagine that in complex architectures, these AIs would also be unlikely to scheme because of other limitations. There are several LLM calls made within part of a complex composite, and each LLM call has very tight information and capability restrictions. Also, we might ensure that any motivation is optimized for the specific request, instead of the LLM aiming to optimize what the entire system does.
2) On the second, I expect that some of this will be pretty natural. Basically, it seems like “LLMs writing code” is already happening, and it seems easy to have creative combinations of LLM agents that write code that they know will be useful for their own reasoning later on. In theory, any function that could either run via an LLM or via interpretable code, should be run via interpretable code. As LLMs get very smart, they might find cleverer ways to write interpretable code that would cover a lot of what LLMs get used for. Over time, composite architectures would rely more and more on this code for reasoning processes. (Even better might be interpretable and proven code)
This might be obvious, but I don’t think we have evidence to support the idea that there really is anything like a concrete plan. All of the statements I’ve seen from Sam on this issue so far are incredibly basic and hand-wavy.
I suspect that any concrete plan would be fairly controversial, so it’s easiest to speak in generalities. And I doubt there’s anything like an internal team with some great secret macrostrategy—instead I assume that they haven’t felt pressured to think through it much.
Agreed. I’m curious how to best do this.
One thing that I’m excited about is using future AIs to judge current ones. So we could have a system that does:
1. An AI today (or a human) would output a certain recommended strategy.
2. In 10 years, we agree to have the most highly-trusted AI evaluator evaluate how strong this strategy was, on some numeric scale. We could also wait until we have a “sufficient” AI, meaning that there might be some set point at which we’d trust AIs to do this evaluation. (I discussed this more here)
3. Going back to ~today, we have forecasting systems predict how well the strategy (1) will do on (2).