I’m currently researching forecasting and epistemics as part of the Quantified Uncertainty Research Institute.
ozziegooen
Karma: 4,225
Submissions end soon (this Sunday)! If there aren’t many, then this can be an easy $300 for someone.
It’s arguably difficult to prove that AIs can be as good or better at moral reasoning than humans.
A lot of the challenge is that there’s no clear standard for moral reasoning. Honestly, I’d guess that a big part of this is that humans are generally quite bad at it, and generally highly overconfident in their own moral intuitions.
But one clearer measure is if AIs can predict human’s moral judgements. Very arguably, if an AI system can predict all the moral beliefs that a human would have after being exposed to different information, then the AI must be capable of doing as good a job at moral reasoning.
There is a very different question that we probably want AIs not to only be able to do moral reasoning as well as humans, but also care about such reasoning. But this is a separate challenge and could be tackled accordingly.
My quick guess is that it would be pretty easy to predict the moral intuitions of many people, with the AI of the next few years or so.
I’d expect it to do well in setting like a test in which many strange/unusual moral settings are described, then humans (of different educational levels and worldviews) need to make judgements.
Develop AIs which are very dumb within a forward pass, but which are very good at using natural language reasoning such that they are competitive with our current systems. Demonstrate that these AIs are very unlikely to be scheming due to insufficient capacity outside of natural language (if we monitor their chains of thought). After ruling out scheming, solve other problems which seem notably easier.
Pursue a very different AI design which is much more modular and more hand constructed (as in, more GOFAI style). This can involve usage of many small and dumb neural components, but needs to be sufficiently interpretable in aggregate which might be hard. This can be done by having the AIs apply huge amounts of labor.
These are two of the main ideas I’m excited about. I’d quickly flag:
1) For the first one, “Demonstrate that these AIs are very unlikely to be scheming due to insufficient capacity outside of natural language ” → I imagine that in complex architectures, these AIs would also be unlikely to scheme because of other limitations. There are several LLM calls made within part of a complex composite, and each LLM call has very tight information and capability restrictions. Also, we might ensure that any motivation is optimized for the specific request, instead of the LLM aiming to optimize what the entire system does.
2) On the second, I expect that some of this will be pretty natural. Basically, it seems like “LLMs writing code” is already happening, and it seems easy to have creative combinations of LLM agents that write code that they know will be useful for their own reasoning later on. In theory, any function that could either run via an LLM or via interpretable code, should be run via interpretable code. As LLMs get very smart, they might find cleverer ways to write interpretable code that would cover a lot of what LLMs get used for. Over time, composite architectures would rely more and more on this code for reasoning processes. (Even better might be interpretable and proven code)
This might be obvious, but I don’t think we have evidence to support the idea that there really is anything like a concrete plan. All of the statements I’ve seen from Sam on this issue so far are incredibly basic and hand-wavy.
I suspect that any concrete plan would be fairly controversial, so it’s easiest to speak in generalities. And I doubt there’s anything like an internal team with some great secret macrostrategy—instead I assume that they haven’t felt pressured to think through it much.
Correct, that wasn’t my intended point. Thanks for clarifying, I’ll try to be more careful in the future.
I partially agree, but I think this must only be a small part of the issue.
- I think there’s a whole lot of key insights people could raise that aren’t info-hazards.
- If secrecy were the main factor, I’d hope that there would be some access-controlled message boards or similar. I’d want the discussion to be intentionally happening somewhere. Right now I don’t really think that’s happening. I think a lot of tiny groups have their own personal ideas, but there’s surprisingly little systematic and private thinking between the power players.
- I think that secrecy is often an excuse not to open ideas to feedback, and thus not be open to critique. Often, what what I see, this goes hand-in-hand with “our work just really isn’t that great, but we don’t want to admit it”
In the last 8 years or so, I’ve kept on hoping there would be some secret and brilliant “master plan” around EA, explaining the lack of public strategy. I have yet to find one. The closest I know of is some over-time discussion and slack threads with people at Constellation and similar—I think these are interesting in terms of understanding the perspectives of these (powerful) people, but I don’t get the impression that there’s all too much comprehensiveness of genius that’s being hidden.
That said,
- I think that policy orgs need to be very secretive, so agree with you regarding why those orgs don’t write more big-picture things.
This is an orthogonal question. I agree that if we’re there now, my claim is much less true.
I’d place fairly little probability mass on this (<10%) and believe much of the rest of the community does as well, though I realize there is a subset of the LessWrong-adjacent community that does.
I’m not sure if it means much, but I’d be very happy if AI safety could get another $50B from smart donors today.
I’d flag that [stopping AI development] would cost far more than $50B. I’d expect that we could easily lose $3T of economic value in the next few years if AI progress seriously stopped.
I guess, it seems to me like duration is basically dramatically more expensive to get than funding, for amounts of funding people would likely want.
Thanks for the specificity!
> On harder-to-operationally-define dimensions (sense of hope and agency for the 25th through 75th percentile of culturally normal people), it’s quite a bit worse.
I think it’s likely that many people are panicking and losing hope each year. There’s a lot of grim media around.
I’m far less sold that something like “civilizational agency” is declining. From what I can tell, companies have gotten dramatically better at achieving their intended ends in the last 30 years, and most governments have generally been improving in effectiveness.
One challenge I’d have for you / others who feel similar to you, is to try to get more concrete on measures like this, and then to show that they have been declining.
My personal guess is that a bunch of people are incredibly anxious over the state of the world, largely for reasons of media attention, and then this spills over into them assuming major global ramifications without many concrete details or empirical forecasts.
In terms of proposing and discussing AI Alignment strategies, I feel like a few individuals have been dominating the LessWrong conversation recently.
I’ve seen a whole lot from John Wentworth and the Redwood team.
After that, it seems to get messier.
There are several individuals or small groups with their own very unique takes. Matthew Barnett, Davidad, Jesse Hoogland, etc. I think these groups often have very singular visions that they work on, that few others have much buy-in with.
Groups like the Deepmind and Anthropic safety teams seem hesitant to write much about or discuss big-picture strategy. My impression is that specific researchers are working typically working on fairly narrow agendas, and that the leaders of these orgs don’t have the most coherent strategies. There’s one big problem that it’s very difficult to be honest and interesting about big-picture AI strategy without saying things that would be bad for a major organization to say.
Most policy people seem focused on policy details. The funders (OP?) seem pretty quiet.
I think there’s occasionally some neat papers or posts that come from AI Policy or groups like Convergence research. But these also don’t seem to be a big part of the conversation I see—like the authors are pretty segmented, and other LessWrong readers and AI safety people don’t pay much attention to their work.
Here are some important-seeming properties to illustrate what I mean:
Robustness of value-alignment: Modern LLMs can display a relatively high degree of competence when explicitly reasoning about human morality. In order for it to matter for RSI, however, those concepts need to also appropriately come into play when reasoning about seemingly unrelated things, such as programming. The continued ease of jailbreaking AIs serves to illustrate this property failing (although solving jailbreaking would not necessarily get at the whole property I am pointing at).
Propagation of beliefs: When the AI knows something, it should know it in a way which integrates well with everything else it knows, rather than easily displaying the knowledge in one context while seeming to forget it in another.
Preference for reasons over rationalizations: An AI should be ready and eager to correct its mistakes, rather than rationalizing its wrong answers. It should be truth-seeking, following thoughts where they lead instead of planning ahead to justify specific answers. It should prefer to valid proof steps over arriving at an answer when the two conflict.
Knowing the limits of its knowledge: Metacognitive awareness of what it knows and what it doesn’t know, appropriately brought to bear in specific situations. The current AI paradigm just has one big text-completion probability distribution, so there’s not a natural way for it to distinguish between uncertainty about the underlying facts and uncertainty about what to say next—hence we get hallucinations.
All of this is more-or-less a version of the metaphilosophy research agenda, framed in terms of current events in AI.
I very much like the concreteness here.
I consider these sorts of things just fundamental epistemic problems, or basic skills that good researchers should have. All superforecasters should be very familiar with issues 2-4, and most probably couldn’t define metaphilosophy. I don’t see the need to be fancy about it.
On that note, I’ll hypothesize that if we were to make benchmarks for any of these items, it would be fairly doable to make AIs that do better than humans on them, then later we could achieve greater and greater measures. I have a hard time imagining tests here that I would feel confident would not get beaten, if there was sufficient money on the line, in the next year or two.
I think that Slop could be a social problem (i.e. there are some communities that can’t tell slop from better content) , but I’m having a harder time imagining it being a technical problem.
I have a hard time imagining a type of Slop that isn’t low in information. All the kinds of Slop I’m familiar with is basically, “small variations on some ideas, which hold very little informational value.”It seems like models like o1 / r1 are trained by finding ways to make information-dense AI-generated data. I expect that trend to continue. If AIs for some reason experience some “slop thresh-hold”, I don’t see how they get much further by using generated data.
I mostly want to point out that many disempowerment/dystopia failure scenarios don’t require a step-change from AI, just an acceleration of current trends.
Do you think that the world is getting worse each year?
My rough take is that humans, especially rich humans, are generally more and more successful.
I’m sure there are ways for current trends to lead to catastrophe—line some trends dramatically increasing and others decreasing, but that seems like it would require a lengthy and precise argument.
In many worlds, if we have a bunch of decently smart humans around, they would know what specific situations “very dumb humans” would mess up, and take the corresponding preventative measures.
A world where many small pockets of “highly dumb humans” could cause an existential catastrophe is one that’s very clearly incredibly fragile and dangerous, enough so that I assume reasonable actors would freak out until it stops being so fragile and dangerous. I think we see this in other areas—like cyber attacks, where reasonable people prevent small clusters of actors from causing catastrophic damage.
It’s possible that the offense/defense balance would dramatically favor tiny groups of dumb actors, and I assume that this is what you and others expect, but I don’t see it yet.
I feel like you’re talking in highly absolutist terms here.
Global wealth is $454.4 trillion. We currently have ~8 Bil humans, with an average happiness of say 6⁄10. Global wealth and most other measures of civilization flourishing that I know of seem to be generally going up over time.
I think that our world makes a lot of mistakes and fails a lot at coordination. It’s very easy for me to imagine that we could increase global wealth by 3x if we do a decent job.
So how bad are things now? Well, approximately, “We have the current world, at $454 Trillion, with 8 billion humans, etc”. To me that’s definitely something to work with.
I assume that current efforts in AI evals and AI interpretability will be pretty useless if we have very different infrastructures in 10 years. For example, I’m not sure how much LLM interp helps with o1-style high-level reasoning.
I also think that later AI could help us do research. So if the idea is that we could do high-level strategic reasoning to find strategies that aren’t specific to specific models/architectures, I assume we could do that reasoning much better with better AI.
Can you explain this position more? I know the bitter lesson, could imagine a few ways it could have implications here.
The second worry is, I guess, a variant of the first: that we’ll use intent-aligned AI very foolishly. That would be issuing a command like “”follow the laws of the nation you originated in but otherwise do whatever you like.” I guess a key consideration in both cases is whether there’s an adequate level of corrigibility.
I’d flag that I suspect that we really should have AI systems forecasting the future and the results of possible requests.
So if people made a broad request like, “follow the laws of the nation you originated in but otherwise do whatever you like”, they should see forecasts for what that would lead to. If there’s any clearly problematic outcomes, those should be apparent early on.
This seems like it would require either very dumb humans, or a straightforward alignment mistake risk failure, to mess up.
A bunch of people in the AI safety landscape seem to argue “we need to stop AI progress, so that we can make progress on AI safety first.”
One flip side to this is that I think it’s incredibly easy for people to waste a ton of resources on “AI safety” at this point.
I’m not sure how much I trust most technical AI safety researchers to make important progress on AI safety now. And I trust most institutions a lot less.
I’d naively expect if any major country would throw $100 Billion on it today, the results would be highly underwhelming. I rarely trust these governments to make progress on concrete technologies with clear progress measures, and “AI Safety” is highly ambiguous and speculative.
As I’ve written about before, I think it’s just hard to know what critical technical challenges will be bottlenecks around AI alignment, given that it’s unclear when this will become an issue or what sorts of architectures we will have then.
All that said, slowing things down seems much safer to me. I assume that at [year(TAI) − 3] we’ll have a decent idea of what’s needed, and extending that duration seems like a safe bet.
I really want to see better strategic discussion about AI safety. If somehow we could spend $10B just to get a better idea of what to actually do, I’d easily suggest that, though strategy is something that’s typically very difficult to spend money on.
Personally, this is one reason why I favor the meta approach of “make better epistemic tools, using AI.” This is an area that can be very concrete and achievable, though it does have its own problems.
Happy to see work to elicit utility functions with LLMs. I think the intersection of utility functions and LLMs is broadly promising.
I want to flag the grandiosity of the title though. “Utility Engineering” sounds like a pretty significant thing. But from what I understand, almost all of the paper is really about utility elicitation (not control, as it spelled out), and it’s really unclear if this represents a breakthrough significant enough for me to feel comfortable with such a name.
I feel like a whole lot of what I see from the Center For AI Safety does this. “Humanity’s Final Exam”? “Superhuman Forecasting”?
I assume that CFAS thinks that CFAS’s work is all pretty groundbreaking and incredibly significant, but I’d kindly encourage names that many other AI safety community members would also broadly agree with going forward.