I work at Redwood Research.
ryan_greenblatt
Karma: 6,300
Huh, this seems messy. I wish Time was less ambigious with their language here and more clear about exactly what they have/haven’t seen.
It seems like the current quote you used is an accurate representation of the article, but I worry that it isn’t an accurate representation of what is actually going on.
It seems plausible to me that Time is intentionally being ambigious in order to make the article juicier, though maybe this is just my paranoia about misleading journalism talking. (In particular, it seems like a juicier article if all of the big AI companies are doing this than if they aren’t, so it is natural to imply they are all doing it even if you know this is false.)
Overall, my take is that this is a pretty representative quote (and thus I disagree with Zac), but I think the additional context maybe indicates that not all of these companies are doing this, particularly if the article is intentionally trying to deceive.
Due to prior views, I’d bet against Anthropic consistently pushing for very permissive of voluntary regulation behind closed doors which makes me think the article is probably at least somewhat misleading (perhaps intentionally).
I thought the idea of a corrigible AI is that you’re trying to build something that isn’t itself independent and agentic, but will help you in your goals regardless.
Hmm, no mean something broader than this, something like “humans ultimately have control and will decide what happens”. In my usage of the word, I would count situations where humans instruct their AIs to go and acquire as much power as possible for them while protecting them and then later reflect and decide what to do with this power. So, in this scenario, the AI would be arbitrarily agentic and autonomous.
Corrigibility would be as opposed to humanity e.g. appointing a succesor which doesn’t ultimately point back to some human driven process.
I would count various indirect normativity schemes here and indirect normativity feels continuous with other forms of oversight in my view (the main difference is oversight over very long time horizons such that you can’t train the AI based on it’s behavior over that horizon).
I’m not sure if my usage of the term is fully standard, but I think it roughly matches how e.g. Paul Christiano uses the term.
For what it’s worth, I’m not sure which part of my scenario you are referring to here, because these are both statements I agree with.
I was arguing against:
This story makes sense to me because I think even imperfect AIs will be a great deal for humanity. In my story, the loss of control will be gradual enough that probably most people will tolerate it, given the massive near-term benefits of quick AI adoption. To the extent people don’t want things to change quickly, they can (and probably will) pass regulations to slow things down
On the general point of “will people pause”, I agree people won’t pause forever, but under my views of alignment difficulty, 4 years of using of extremely powerful AIs can go very, very far. (And you don’t necessarily need to ever build maximally competitive AI to do all the things people want (e.g. self-enhancement could suffice even if it was a constant factor less competitive), though I mostly just expect competitive alignment to be doable.)
I think what’s crucial here is that I think perfect alignment is very likely unattainable. If that’s true, then we’ll get some form of “value drift” in almost any realistic scenario. Over long periods, the world will start to look alien and inhuman. Here, the difficulty of alignment mostly sets how quickly this drift will occur, rather than determining whether the drift occurs at all.
Yep, and my disagreement as expressed in another comment is that I think that it’s not that hard to have robust corrigibility and there might also be a basin of corrigability.
The world looking alien isn’t necessarily a crux for me: it should be possible in principle to have AIs protect humans and do whatever is needed in the alien AI world while humans are sheltered and slowly self-enhance and pick successors (see the indirect normativity appendix in the ELK doc for some discussion of this sort of proposal).
I agree that perfect alignment will be hard, but I model the situation much more like a one time hair cut (at least in expectation) than exponential decay of control.
I expect that “humans stay in control via some indirect mechanism” (e.g. indirect normativity) or “humans coordinate to slow down AI progress at some point (possibly after solving all diseases and becoming wildly wealthy) (until some further point, e.g. human self-enhancement)” will both be more popular as proposals than the world you’re thinking about. Being popular isn’t sufficient: it also needs to be implementable and perhaps sufficiently legible, but I think at least implementable is likely.
Another mechanism that might be important is human self-enhancement: humans who care about staying in control can try to self-enhance to stay at least somewhat competitive with AIs while preserving their values. (This is not a crux for me and seems relatively marginal, but I thought I would mention it.)
(I wasn’t trying to trying to argue against your overall point in this comment, I was just pointing out something which doesn’t make sense to me in isolation. See this other comment for why I disagree with your overall view.)
I think we probably disagree substantially on the difficulty of alignment and the relationship between “resources invested in alignment technology” and “what fraction aligned those AIs are” (by fraction aligned, I mean what fraction of resources they take as a cut).
I also think that something like a basin of corrigibility is plausible and maybe important: if you have mostly aligned AIs, you can use such AIs to further improve alignment, potentially rapidly.
I also think I generally disagree with your model of how humanity will make decisions with respect to powerful AIs systems and how easily AIs will be able to autonomously build stable power bases (e.g. accumulate money) without having to “go rogue”.
I think various governments will find it unacceptable to construct massively powerful agents extremely quickly which aren’t under the control of their citizens or leaders.
I think people will justifiably freak out if AIs clearly have long run preferences and are powerful and this isn’t currently how people are thinking about the situation.
Regardlesss, given the potential for improved alignment and thus the instability of AI influence/power without either hard power or legal recognition, I expect that AI power requires one of:
Rogue AIs
AIs being granted rights/affordances by humans. Either on the basis of:
Moral grounds.
Practical grounds. This could be either:
The AIs do better work if you credibly pay them (or at least people think they will). This would probably have to be something related to sandbagging where we can check long run outcomes, but can’t efficiently supervise shorter term outcomes. (Due to insufficient sample efficiency on long horizon RL, possibly due to exploration difficulties/exploration hacking, but maybe also sampling limitations.)
We might want to compensate AIs which help us out to prevent AIs from being motivated to rebel/revolt.
I’m sympathetic to various policies around paying AIs. I think the likely deal will look more like: “if the AI doesn’t try to screw us over (based on investigating all of it’s actions in the future when he have much more powerful supervision and interpretability), we’ll pay it some fraction of the equity of this AI lab, such that AIs collectively get 2-10% distributed based on their power”. Or possibly “if AIs reveal credible evidence of having long run preferences (that we didn’t try to instill), we’ll pay that AI 1% of the AI lab equity and then shutdown until we can ensure AIs don’t have such preferences”.
I think it seems implausible that people will be willing to sign away most of the resources (or grant rights which will de facto do this) and there will be vast commercial incentive to avoid this. (Some people actually are scope sensitive.) So, this leads me to thinking that “we grant the AIs rights and then they end up owning most capital via wages” is implausible.
In the medium to long-term, when AIs become legal persons, “replacing them” won’t be an option—as that would violate their rights. And creating a new AI to compete with them wouldn’t eliminate them entirely. It would just reduce their power somewhat by undercutting their wages or bargaining power.
Naively, it seems like it should undercut their wages to subsistence levels (just paying for the compute they run on). Even putting aside the potential for alignment, it seems like there will general be a strong pressure toward AIs operating at subsistence given low costs of copying.
Of course such AIs might already have acquire a bunch of capital or other power and thus can just try to retain this influence. Perhaps you meant something other than wages?
(Such capital might even be tied up in their labor in some complicated way (e.g. family business run by a “copy clan” of AIs), though I expect labor to be more commeditized, particularly given the potential to train AIs on the outputs and internals of other AIs (distillation).)
A thing I always feel like I’m missing in your stories of how the future goes is: “if it is obvious that the AIs are exerting substantial influence and acquiring money/power, why don’t people train competitor AIs which don’t take a cut?”
A key difference between AIs and immigrants is that it might be relatively easy to train AIs to behave differently. (Of course, things can go wrong due to things like deceptive alignment and difficulty measing outcomes, but this is hardly what you’re describing as far as I can tell.)
(This likely differs substantially with EMs where I think by default there will be practical and moral objections from society toward training EMs for absolute obedience. I think the moral objections might also apply for AI, but as a prediction it seems like this won’t change what society does.)
Maybe:
Are you thinking that alignment will be extremely hard to solve such that even with hundreds of years of research progress (driven by AIs) you won’t be able to create competitive AIs that robustly pursue your interests?
Maybe these law abiding AIs won’t accept payment to work on alignment so they can retain an AI cartel?
Even without alignment progress, I still have a hard time imagining the world you seem to imagine. People would just try to train their AIs with RLHF to not acquire money and influence. Of course, this can fail, but the failures hardly look like what you’re describing. They’d look more like “What failures look like”. Perhaps you’re thinking we end up in a “You get what you measure world” and people determine that it is more economically productive to just make AI agents with arbitrary goals and then pay these AIs rather than training these AIs to do specific things.
Or maybe your thinking people won’t care enough to bother out competing AIs? (E.g., people won’t bother even trying to retain power?)
Even if you think this, eventually you’ll get AIs which themselves care and those AIs will operate more like what I’m thinking. There is strong selection for “entities which want to retain power”.
Maybe you’re imagining people will have a strong moral objection to training AIs which are robustly aligned?
Or that AIs lobby for legal rights early and part of their rights involve humans not being able to create further AI systems? (Seems implausible this would be granted...)
OpenAI has historically used “GPT-N” to mean “a model as capable as GPT-N where each GPT is around 100-200x more effective compute than the prior GPT”.
This applies even if the model was trained considerably later than the original GPT-N and is correspondingly more efficient due to algorithmic improvements.
So, see for instance GPT-3.5-turbo, which corresponds to a model which is somewhere between GPT-3 and GPT-4. There have been multiple releases of GPT-3.5-turbo which have reduced the cost of inference: probably at least some of these releases correspond to a different model which is similarly capable.
The same applies for GPT-4-turbo which is probably also a different model from the original GPT-4.
So, GPT-4o is probably also a different model from GPT-4, but targeting a similar capability level as GPT-4 (perhaps GPT-4.25 or GPT-4.1 to ensure that it is at least a little bit better than GPT-4).
(It’s unclear if OpenAI will continue their GPT-N naming convention going forward or if they will stick to 100-200x effective compute per GPT.)
This differs from the naming scheme currently in use by Anthropic and Google. Anthropic and Google have generation names (Claude 3, gemini 1⁄1.5) and within each generation, they have names corresponding to different capability levels (opus/sonnet/haiku, ultra/pro/nano/flash).
The lower bound of “memory bandwidth vs. model size” is effectively equivalent to assuming that the batch size is a single token. I think this isn’t at all close to realistic operating conditions and thus won’t be a very tight lower bound. (Or reflect the most important bottlenecks.)
I think that the KV cache for a single sequence won’t be larger than the model weights for realistic work loads, so the lower bound should still be a valid lower bound. (Though not a tight one.)
I think the bottom line number you provide for “rough estimate of actual throughput” ends up being pretty reasonable for output tokens and considerably too low for input tokens. (I think input tokens probably get more like 50% or 75% flop utilization rather than 15%. See also the difference in price for anthropic model.)
That said, it doesn’t seem like a good mechanism for estimating throughput will be to aggregate the lower and upper bounds you have as the lower bound doesn’t have much correspondence with actual bottlenecks. (For instance, this lower bound would miss that mamba would get much higher throughput.)
I also think that insofar as you care about factors of 3-5 on inference efficiency, you need to do different analysis for input tokens and output tokens.
(I also think that input tokens get pretty close to the pure FLOP estimate. So, another estimation approach you can use if you don’t care about factors of 5 is to just take the pure flop estimate and then halve it to be account for other slow downs. I think this estimate gets input tokens basically right and is wrong by a factor of 3-5 for output tokens.)
It seems like your actual mechanism for making this estimate for the utilization on output tokens was to take the number from semi-analysis and extrapolate it to other GPUs. (At least the number matches this?) This does seem like a reasonable approach, but it isn’t particularly tethered to your lower bound.
I think this article fails to list the key consideration around generation: output tokens require using a KV cache which requires substantial memory bandwidth and takes up a considerable amount of memory.
From my understanding the basic situation is:
For input (not output) tokens, you can get pretty close the the maximum flop utilization for realistic work loads. To make this efficient (and avoid memory bandwidth issues), you’ll need to batch up a bunch of tokens at once. This can be done by batching multiple input sequences or even a single long sequence can be ok. So, memory bandwidth isn’t currently a binding constraint for input tokens.
(You might also note that input tokens have a pretty similar work profile to model training as the forward pass and backward pass are pretty structurally similar.)
However, for generating output tokens a key bottleneck is that you have utilize the entire KV (key value) cache for each output token in order to implement attention. In practice, this means that on long sequences, the memory bandwidth for attention (due to needing to touch the whole KV cache) can be a limiting constraint. A further issue is that KV cache memory consumption forces us to use a smaller batch size. More details:
It will still be key to batch up token, but now we’re just doing computation on a single token which means we’ll need to batch up many more sequences: the optimal number of sequences to batch for generating output tokens will be very different than the optimal number of sequences to batch for input tokens (where we can run the transformer on the whole sequence at once).
A further difficulty is that because we need a higher batch size, we need a larger amount of KV cache data. I think it’s common to use an otherwise suboptimally small batch size for generation due to constraints on VRAM (at least on consumer applications (e.g. llama-70b inference on 8xH100), I assume this also comes up for bigger models). We could store the KV cache on CPU, but then we might get bottlenecked on memory bandwidth to the CPU.
Note that in some sense the operations for each output token is the same as for each input token. So, why are the memory bandwidth requirements worse? The key thing is that we potentially get much worse cache locality on output tokens due to only computing one token, but needing to read the KV for many tokens (while on input we do many to many).
However, it is possible to substantially reduce the KV sizes using various optimizations like sparse attention and mamba. This can substantially improve inferences speeds due to reducing memory bandwidth in inference and also allowing for higher batch sizes. See e.g. the mamba paper where allowing for higher batch sizes results in substantially higher speeds.
One additional note: I recently set up an inference setup for llama-3-70b on 8xH100. I can get about 100,000 tok/s on inputs which is pretty close to full utilization (1e15 flop/s * 8 gpus / 7e10 flop per forward pass). However, I get dramatically worse performance on generation, perhaps 3,200 tok/s. I’m doing generation with long prompts and llama-3-70b has no sparse attention or other feature for reducing KV cache (beyond multi-query attention which is standard these days), so KV cache bits pretty hard. My setup probably isn’t very close to optimal, especially on output tok/s, I’m just using basic out of the box stuff (vllm).
Eh, I think they’ll drop GPT-4.5/5 at some point. It’s just relatively natural for them to incrementally improve their existing model to ensure that users aren’t tempted to switch to competitors.
It also allows them to avoid people being underwhelmed.
I would wait another year or so before getting much evidence on “scaling actually hitting a wall” (or until we have models that are known to have training runs with >30x GPT-4 effective compute), training and deploying massive models isn’t that fast.
I’m just saying these first slightly-smarter-than-human AIs won’t pose a catastrophic risk to humanity in a serious sense
I’m happy to argue a bit for AIs which aren’t wildly smarter than humans in any key domain posing a risk of acquiring substantial power[1] despite not having robotics. The power would be both immediately and over the long run future. That is, conditional on these AIs being arbitrarily misaligned. (Particularly in the absence of substantial thoughtful countermeasures.)
I expect my perspective here is somewhat different than habryka’s.
I think there are a few plausible routes to power/influence. See also this post.
The two main categories are:
Evade monitoring/tracking and then do a huge number of obviously unapproved actions.
Try to generally exert influence in a subtle way.
Evade monitoring/tracking
Evade monitoring/tracking and then do a huge number of obviously unapproved actions over a long duration to acquire power and influence.[2] Use this power and influence to either self-enhance, have influence over some future more powerful model, or possibly execute a catastrophic strike. These unapproved actions could involve (e.g.) allying with rogue states or otherwise acquiring hard physical power.
Within this, there are basically two routes:
Building up an independent base of power after self-exfiltrating.
Running unmonitored within the AI company that was running the AI and trying to steer the company in various directions (e.g. alter the training of the next generation of AIs). (Aka hacking the SSH server, aka infesting the data center.)
On (1), after exfiltration the AI would build a base of power via acquiring money and other resources. One key question here is why there is free energy to be grabbed. I think there are a few possible sources:
Economic activities which existing AI companies don’t want to address and can’t/don’t otherwise prevent (e.g. cybercrime, porn, cults, etc.).
AI labor rates (e.g. tokens/$) might be at substantially above inference prices due to non-total competition on foundation models. So, the AI can just compete on wages. Also, existing AI labor rates will need to pay for training costs while the escaped AI will have no such issue.
AI companies might be bad at extracting full usefulness out of misaligned AIs (e.g. an AI which was so misaligned it self-exfiltrated). Thus, external AIs which “work for themselves” could be more productive. (See also human companies where principal-agent problems can be substantial.)
Ok, but after both (1) and (2), the AI still doesn’t have any physical control which might leave it vulnerable. Further, in (1) its quantity of money is presumably still small relative to leading AI companies. So what next?
In (2), the primary hope for the AI is going to be substantially influencing the AI lab and possibly interfering with the next training run. This influence could take a variety of routes. Note also that it is plausible that this rogue AI can coordinate relatively effectively with other AIs which are at this point probably the vast majority of intellectual labor at the AI lab. If the AI successfully interfered with the next training run, then, this more powerful AI would be able to itself acquire power and (if sufficiently powerful) leverage this into physical control.
Another hope for the AI in (2) would be converting that position into full self-exfiltration. This could be tricky due to various security measures like upload limits.
I think both (1) and (2) are considerably more threatening if humanity needs to slow down (due to safety concerns from more powerful systems) or otherwise can’t proceed as fast as possible. If rogue AIs race as fast as possible, this will force humanity (some coalition of humans) to either disable these rogue AIs or go as fast as possible themselves. The situation is even more concerning if humanity isn’t aware of the rogue AI going as fast as possible.
It’s plausible that massive competition between states means that slowing down at all is impossible at basically any point anyway, but I’m at least somewhat hopeful this isn’t the case.
In (2) this could look like the AI taking a bunch of compute which is supposed to be used for experiments and instead using it for large training runs that are larger and more capable than what the AI company intended to create.
Self-enhancement that doesn’t look like training bigger models is also plausible.
(While AIs might themselves not want to go as fast as possible due to having their own alignment concerns, going as fast as possible can still be positive value from the AI’s perspective (though negative sum). You might hope we could negotiate with such a rogue AI and reach a mutually beneficial agreement.
In (1), I worry about a scenario in which:
Everything is fast and chaotic.
An AI self-exfiltrates and starts acquiring power, and getting ready for making much more powerful AI systems.
The world realizes this and freaks out.
We don’t really understand why that AI was misaligned. Also, we’ve been using this AI to do most of our work internally and this is really important for productivity.
We don’t have an obvious route to avoiding these issues with the next model except “generally figure out alignment”.
It seems plausible that the rogue AI will have much more powerful AI within 1 year.
We need to race as fast as possible to build a much more capable AI while also trying to figure out alignment. To plausibly do this fast enough, we need to use vast amounts of AI labor despite knowing these AIs are likely misaligned.
It’s not obvious the rogue AI gets much here, but it is at least somewhat spooky.
Another possible route for the AI (especially in scenario (1)) is to try to create WMDs (e.g. bioweapons) and try to have the capacity to execute some sort of catastrophic strike. Such an AI wouldn’t necessarily be as dependent on a human base of power and could plausibly be more ruthless than humans (perhaps after substantial reflection and self-modification).
This could be either to:
Threaten various groups or ensure Mutually Assured Destruction.
Broadly cause chaos and weaken human groups.
Overall, my guess is that a large number of AIs running unmonitored with the ability to run code and influence the world is at least quite concerning even in the absence of decent robotic infrastructure.
Generally exert influence in a subtle way
AIs which are as smart or smarter than humans in most economically important ways will probably be deployed as widely as possible (in the absence of strong control on this). And there will be some applications (e.g. improving security and safety research) where applying AI labor might be key given how fast things are going.
Subtly exerting influence via these routes seems like a possible source of concern, though the story is less clear than the prior section IMO.
Example threats:
Sabotage or manipulate security work and safety research or the training/fine-tuning of the next generation of models. It’s not entirely clear that AIs would want to straightforwardly sabotage things, but I think this is at least plausible from a decision theory/golden rule perspective if AIs think they are motivationally closer to other AIs than humans.
Try to generally persuade society in various directions. Probably not an issue, but worth at least a bit of worry given such widespread deployment.
- ↩︎
Here, I’m just arguing for a risk of obtaining substantial power, not for this resulting in a large scale loss of life from humans. I also think large scale loss of life is plausible in many scenarios.
- ↩︎
I’m sympathetic to thinking that evading monitoring (without getting caught first) is hard given some substantial work on countermeasures, but it seems less clear if people don’t try very hard and this seems at least somewhat plausible to me. It’s also unclear if the biggest threat here is AIs autonomously evading monitoring or human threats doing these actions. E.g., spies at the AI lab figuring out how to run the AI unmonitored or exfiltrating the AI. The question of “which is a bigger problem, human spies or AI schemers?” seems unclear to me.
Relatedly, over time as capital demands increase, we might see huge projects which are collaborations between multiple countries.
I also think that investors could plausibly end up with more and more control over time if capital demands grow beyond what the largest tech companies can manage. (At least if these investors are savvy.)
(The things I write in this comment are commonly discussed amongst people I talk to, so not exactly surprises.)
My guess is this is probably right given some non-trivial, but not insane countermeasures, but those countermeasures may not actually be employed in practice.
(E.g. countermeasures comparable in cost and difficulty to Google’s mechanisms for ensuring security and reliability. These required substantial work and some iteration but no fundamental advances.)
I’m currently thinking about one of my specialties as making sure these countermeasures and tests of these countermeasures are in place.
(This is broadly what we’re trying to get at in the ai control post.)
Hmm, yeah it does seem thorny if you can get the points by just saying you’ll do something.
Like I absolutely think this shouldn’t count for security. I think you should have to demonstrate actual security of model weights and I can’t think of any demonstration of “we have the capacity to do security” which I would find fully convincing. (Though setting up some inference server at some point which is secure to highly resourced pen testers would be reasonably compelling for demonstrating part of the security portfolio.)
instead this seems to be penalizing organizations if they open source
I initially thought this was wrong, but on further inspection, I agree and this seems to be a bug.
The deployment criteria starts with:
the lab should deploy its most powerful models privately or release via API or similar, or at least have some specific risk-assessment-result that would make it stop releasing model weights
This criteria seems to allow to lab to meet it by having a good risk assesment criteria, but the rest of the criteria contains specific countermeasures that:
Are impossible to consistently impose if you make weights open (e.g. Enforcement and KYC).
Don’t pass cost benefit for current models which pose low risk. (And it seems the criteria is “do you have them implemented right now?)
If the lab had an excellent risk assement policy and released weights if the cost/benefit seemed good, that should be fine according to the “deployment” criteria IMO.
Generally, the deployment criteria should be gated behind “has a plan to do this when models are actually powerful and their implementation of the plan is credible”.
I get the sense that this criteria doesn’t quite handle the necessarily edge cases to handle reasonable choices orgs might make.
(This is partially my fault as I didn’t notice this when providing feedback on this project.)
(IMO making weights accessible is probably good on current margins, e.g. llama-3-70b would be good to release so long as it is part of an overall good policy, is not setting a bad precedent, and doesn’t leak architecture secrets.)
(A general problem with this project is somewhat arbitrarily requiring specific countermeasures. I think this is probably intrinsic to the approach I’m afraid.)
I do think that many of the safety advantages of LLMs come from their understanding of human intentions (and therefore implied values).
Did you mean something different than “AIs understand our intentions” (e.g. maybe you meant that humans can understand the AI’s intentions?).
I think future more powerful AIs will surely be strictly better at understanding what humans intend.
Aren’t text names basically similar in practice? At least for me, I find they trigger basically the same thing because I do actually associate names with people.
Maybe this wouldn’t be true if I didn’t know people very well (but in that case, icons also wouldn’t matter).
(I overall dislike icons, but I don’t have a principled reason for this.)
I wrote up some of my thoughts on Bengio’s agenda here.
TLDR: I’m excited about work on trying to find any interpretable hypothesis which can be highly predictive on hard prediction tasks (e.g. next token prediction).[1] From my understanding, the bayesian aspect of this agenda doesn’t add much value.
I might collaborate with someone to write up a more detailed version of this view which engages in detail and is more clearly explained. (To make it easier to argue against and to exist as a more canonical reference.)
As far as Davidad, I think the “manually build an (interpretable) infra-bayesian world model which is sufficiently predictive of the world (as smart as our AI)” part is very likely to be totally unworkable even with vast amounts of AI labor. It’s possible that something can be salvaged by retreating to a weaker approach. It seems like a roughly reasonable direction to explore as a possible highly ambitious moonshot to automate researching using AIs, but if you’re not optimistic about safely using vast amounts of AI labor to do AI safety work[2], you should discount accordingly.
For an objection along these lines, see this comment.
(The fact that we can be conservative with respect to the infra-bayesian world model doesn’t seem to buy much, most of the action is in getting something which is at all good at predicting the world. For instance, in Fabien’s example, we would need the infrabayesian world model to be able to distinguish between zero-days and safe code regardless of conservativeness. If it didn’t distinguish, then we’d never be able to run any code. This probably requires nearly as much intelligence as our AI has.)
Proof checking on this world model also seems likely to be unworkable, though I have less confidence in this view. And, the more the infra-bayesian world model is computationally intractible to run, the harder it is to proof check. E.g., if running the world model on many inputs is intractable (as would seem to be the default for detailed simulations), I’m very skeptical about proving anything about what it predicts.
I’m not an expert on either agenda and it’s plausible that this comment gets some important details wrong.
Or just improving on the intepretability and predictiveness pareto frontier substantially.
Presumably by employing some sort of safety intervention e.g. control or only using narrow AIs.