The following is an edited partial chat transcript of a conversation involving me, Benquo, and an anonymous person (X). I am posting it in the hope that it has enough positive value-of-information compared to the attentional cost to be of benefit to others. I hope people can take this as “yay, I got to witness a back-room conversation” rather than “oh no, someone has made a bunch of public assertions that they can’t back up”; I think it would be difficult and time-consuming to argue for all these points convincingly, although I can explain to some degree, and I’m not opposed to people stating their disagreements. There were concerns about reputational risk as a result of posting this, which is partially mitigated by redaction, and partially is a cost that is intentionally “eaten” to gain the value-of-information benefit.
X: [asking why Jess is engaging with the post]
Jess: It seems like there’s enough signal in the post that, despite the anxious time-wasting, it was worth contemplating. It’s good that Eliezer is invalidating people’s strategy of buying insurance products for AI catastrophe from a secretive technocratic class centered around him. Same as the conclusion I reached in 2017. I just hope that people can generalize from “alignment is hard” to “generalized AI capabilities are hard”.
X: I agree it was worth reading… I just worry about the people in the bubble who are dependent on it. I think people should be trying to get people out of the bubble and trying to deflate it so that if (when) it collapses it has less of a negative impact. (Bubble = narrative bubble.) Engaging with it directly in a way that buys its assumption fuels the bubble, rather than reducing it.
Jess: [REDACTED] talked about a bubble at MIRI popping earlier last year having to do with their basic assumptions about what they can accomplish being invalidated. The problem is they’re still promoting the short AI timelines → doom bubble. They popped their bubble about having special access to solutions. Part of what’s going on is that the old bubble ran on a combination of making people scared about AI and giving them an opportunity to offload responsibility for risk onto a techocratic class. Eliezer is invalidating the second part of this but not the first. There are a bunch of upset emotional reactions in the comments about this. So this seems like an “evaporative cooling” moment in Eliezer’s terminology.
X: The thing is centered around Eliezer. As far as I understand it, the argument for doom is “someone thought of something Eliezer thought only he had thought of, so Eliezer hard-updated to it not being that hard, so, short timelines.”
Jess: Yeah, that was one thing that caused his update, although there were other sources too.
X: Yes, there’s info from Demis, Dario, the OpenPhil person, and Gwern. But that stuff is not tracked especially precisely, as far as I understand it, so I don’t think those are the driving factors.
Ben: I think there’s a serious confusion around how tightly coupled the ability to access large amounts of capital and spend it on huge amounts of compute to create special-case solutions for well-contained games is to the interest in using thinking to solve problems rather than perpetuate the grift. Relatedly, [REDACTED] told me a couple months ago that his speech-recognition startup was able to achieve higher levels of accuracy than the competition by hiring (cheaper) linguists and not just (more expensive but prestigious) machine learning programmers, but he was surprised to find that later speech-recognition startups didn’t just copy what worked, but instead spent all their $ on ML people and compute. This is seriously confusing to someone coming from Eliezer’s perspective, I made the same error initially when deep learning became a thing. Debtors’ Revolt was my attempt to put together a structural explanation.
X: This doesn’t surprise me but it’s great to have the data point. I looked into speech recognition a little while ago and concluded ML was likely not what was needed.
Ben: Eliezer thinks that OpenAI will try to make things go faster rather than slower, but this is plainly inconsistent with things like the state of vitamin D research or, well, most of academia & Parkinson’s Law more generally. Demis & Elon ended up colluding to create a Basilisk, i.e. a vibe of “the UFAI is big and dangerous so you must placate it by supporting attempts to build it”. It’s legitimately confusing to make sense of, especially if your most trusted advisors are doing some combination of flattering your narcissistic fantasy of specialness, and picking up a party line from OpenAI.
X: That’s why I put the cause at Eliezer’s failure to actually admit defeat. There are reasons, etc., but if you chase it upstream you hit things like “too lazy to count” or “won’t pay attention to something” or “someone thought of a thing he thought of” etc. I think that if Eliezer were more honest, he would update down on “rationality is systematized winning”, admit that, and then admit that his failure is evidence his system is corrupted.
Jess: I don’t really understand this. In a sense he is admitting defeat?
X: He’s not admitting defeat. He’s saying he was beaten by a problem that was TOO hard. Here’s a way to admit defeat: “wow I thought I was smart and smart meant could solve problems but I couldn’t solve the problems MAYBE I’M NOT ACTUALLY THAT SMART or maybe something else has gone wrong, hey, I think my worldview was just undermined”
Jess: I see what you mean, there’s some failure to propagate a philosophical update.
X: On Eliezer’s view, he’s still the smartest, most rational human ever. (Or something.)
Ben: He’s also not trying to share the private info he alludes to in lieu of justifying short timelines, even though he knows he has no idea what to do with it and can’t possibly know that no one else does.
Jess: So, as I wrote on Twitter recently, systematized winning would be considered stage 2 in Kegan’s model, and Kegan says “rationality” is stage 4, which is a kind of in-group-favoring (e.g. religious, nationalistic) “doing one’s duty” in a system.
X: Well, he can’t share the private short timelines info because there really isn’t any. This was a big update I took from you/my/Jessica’s conversation. I had been assuming that MIRI et al had private info, but then I came to believe they were just feeding numbers to each other, which in my universe does not count.
Ben: He’s also not sharing that.
X: Right, this is why I think the thing is not genuine. He knows that if he said “Anna, Nate, and I sat in a circle and shared numbers, and they kept going down, especially after I realized that someone else had an insight that I had had, though also important were statements from some other researchers, who I’m relying on rather than having my own view” people would 🤨🤨🤨 and that would diminish his credibility.
Jess: OpenAI has something of a model for short timelines, which is like Median’s AI compute model but with parameters biased towards short timelines relative to Median’s. However I think Eliezer and Nate think it’s more insight driven, and previously thought they personally possessed some relevant insights.
X: This nuances my model, thanks
Jess: But they don’t share them because “danger” so no one can check their work, and it looks like a lot of nothing from the outside.
I began reading this charitably (unaware of whatever inside baseball is potentially going on, and seems to be alluded to), but to be honest struggled after “X” seemed to really want someone (Eliezer) to admit they’re “not smart”? I’m not sure why that would be relevant.
I think I found these lines especially confusing, if you want to explain:
“I just hope that people can generalize from “alignment is hard” to “generalized AI capabilities are hard”.
Is capability supposed to be hard for similar reasons as alignment? Can you expand/link? The only argument I can think of relating the two (which I think is a bad one) is “machines will have to solve their own alignment problem to become capable.”
Eliezer is invalidating the second part of this but not the first.
This would be a pretty useless machiavellian strategy, so I’m assuming you’re saying it’s happening for other reasons? Maybe self-deception? Can you explain?
Eliezer thinks that OpenAI will try to make things go faster rather than slower, but this is plainly inconsistent with things like the state of vitamin D research
This just made me go “wha” at first but my guess now is that this and the bits above it around speech recognition seem to be pointing at some AI winter-esque (or even tech stagnation) beliefs? Is this right?
I began reading this charitably (unaware of whatever inside baseball is potentially going on, and seems to be alluded to), but to be honest struggled after “X” seemed to really want someone (Eliezer) to admit they’re “not smart”? I’m not sure why that would be relevant.
I’m not sure exactly what is meant, one guess is that it’s about centrality: making yourself more central (more making executive decisions, more of a bottleneck on approving things, more looked to as a leader by others, etc) makes more sense the more you’re more correct about relevant things relative to other people. Saying “oh, I was wrong about a lot, whoops” is the kind of thing someone might do before e.g. stepping down as project manager or CEO. If you think your philosophy has major problems and your replacements’ philosophies have fewer major problems, that might increase the chance of success.
I would guess this is comparable to what Eliezer is saying in this post about how some people should just avoid consequentialist reasoning because they’re too bad at it and unlikely to improve:
People like this should not be ‘consequentialists’ or ‘utilitarians’ as they understand those terms. They should back off from this form of reasoning that their mind is not naturally well-suited for processing in a native format, and stick to intuitively informally asking themselves what’s good or bad behavior, without any special focus on what they think are ‘outcomes’.
If they try to be consequentialists, they’ll end up as Hollywood villains describing some grand scheme that violates a lot of ethics and deontology but sure will end up having grandiose benefits, yup, even while everybody in the audience knows perfectly well that it won’t work. You can only safely be a consequentialist if you’re genre-savvy about that class of arguments—if you’re not the blind villain on screen, but the person in the audience watching who sees why that won’t work.
...
Is capability supposed to be hard for similar reasons as alignment? Can you expand/link? The only argument I can think of relating the two (which I think is a bad one) is “machines will have to solve their own alignment problem to become capable.”
Alignment is hard because it’s a quite general technical problem. You don’t just need to make the AI aligned in case X, you also have to make it aligned in cases Y and Z. To do this you need to create very general analysis and engineering tools that generalize across these situations.
Similarly, AGI is a quite general technical problem. You don’t just need to make an AI that can do narrow task X, it has to work in cases Y and Z too, or it will fall over and fail to take over the world at some point. To do this you need to create very general analysis and engineering tools that generalize across these situations.
For an intuition pump about this, imagine that LW’s effort towards making an aligned AI over the past ~14 years was instead directed at making AGI. We have records of certain mathematical formalisms people have come up with (e.g. UDT, logical induction). These tools are pretty far from enhancing AI capabilities. If the goal had been to enhance AI capabilities, they would have enhanced AI capabilities more, but still, the total amount of intellectual work that’s been completed is quite small compared to how much intellectual work would be required to build a working agent that generalizes across situations. The AI field has been at this for decades and has produced the results that it has, which are quite impressive in some domains, but still fail to generalize most of the time, and even what has been produced has required a lot of intellectual work spanning multiple academic fields and industries over decades. (Even if the field is inefficient in some ways, that would still imply that inefficiency is common, and LW seems to also be inefficient at solving AI-related technical problems compared to its potential.)
This would be a pretty useless machiavellian strategy, so I’m assuming you’re saying it’s happening for other reasons? Maybe self-deception? Can you explain?
I’m not locating all the intentionality for creating these bubbles in Eliezer, there are other people in the “scene” that promote memes and gain various benefits from them (see this dialogue, ctrl-f “billions”).
There’s a common motive to try to be important by claiming that one has unique skills to solve important problems, and pursuing that motive leads to stress because it involves creating implicit or explicit promises that are hard to fulfil (see e.g. Elizabeth Holmes), and telling people “hey actually, I can’t solve this” reduces the stress level and makes it easier to live a normal life.
This just made me go “wha” at first but my guess now is that this and the bits above it around speech recognition seem to be pointing at some AI winter-esque (or even tech stagnation) beliefs? Is this right?
I think what Ben means here is that access to large amounts of capital is anti-correlated with actually trying to solve difficult intellectual problems. This is the opposite of what would be predicted by the efficient market hypothesis.
The Debtors’ Revolt argues that college (which many, many more Americans have gone too than previously) primarily functions to cause people to correlate with each other, not to teach people epistemic and instrumental rationality. E.g. college educated people are more likely to immediately dismiss Vitamin D as a COVID health intervention (due to an impression of “expert consensus”) rather than forming an opinion based on reading some studies and doing probability calculations. One would by default expect epistemic/instrumental rationality to be well-correlated with income, for standard efficient market hypothesis reasons. However, if there is a massive amount of correlation among the “irrational” actors, they can reward each other, provide insurance to each other, commit violence in favor of their class (e.g. the 2008 bailouts), etc.
(On this model, a major reason large companies do the “train a single large, expensive model using standard techniques like transformers” is to create correlation in the form of a canonical way of spending resources to advance AI.)
Similarly, AGI is a quite general technical problem. You don’t just need to make an AI that can do narrow task X, it has to work in cases Y and Z too, or it will fall over and fail to take over the world at some point. To do this you need to create very general analysis and engineering tools that generalize across these situations.
I don’t think this is a valid argument. Counter-example: you could build an AGI by uploading a human brain onto an artificial substrate, and you don’t “need to create very general analysis and engineering tools that generalize across these situations” to do this.
More realistically, it seems pretty plausible that all of the necessary patterns/rules/heuristics/algorithms/forms of reasoning necessary for “being generally intelligent” can be found in human culture, and ML can distill these elements of general intelligence into a (language or multimodal) model that will then be generally intelligent. This also doesn’t seem to require very general analysis and engineering tools. What do you think of this possibility?
You’re right that the uploading case wouldn’t necessarily require strong algorithmic insight. However, it’s a kind of bounded technical problem that’s relatively easy to evaluate progress in relative to the difficulty, e.g. based on ability to upload smaller animal brains, so would lead to >40 year timelines absent large shifts in the field or large drivers of progress. It would also lead to a significant degree of alignment by default.
For copying culture, I think the main issue is that culture is a protocol that runs on human brains, not on computers. Analogously, there are Internet protocols saying things like “a SYN/ACK packet must follow a SYN packet”, but these are insufficient for understanding a human’s usage of the Internet. Copying these would lead to imitations, e.g. machines that correctly send SYN/ACK packets and produce semi-grammatical text but lack certain forms of understanding, especially connection to a surrounding “the real world” that is spaciotemporal etc.
If you don’t have logic yourself, you can look at a lot of logical content (e.g. math papers) without understanding logic. Most machines work by already working, not by searching over machine designs that fit a dataset.
Also in the cultural case, if it worked it would be decently aligned, since it could copy cultural reasoning about goodness. (The main reason I have for thinking cultural notions of goodness might be undesirable is thinking that, as stated above, culture is just a protocol and most of the relevant value processing happens in the brains, see this post.)
Jess: But they don’t share them because “danger” so no one can check their work, and it looks like a lot of nothing from the outside.
X: It’s a shocking failure of rationality.
Jess: Yes.
There’s an awkward issue here, which is: how can there be people who are financially supported to do research on stuff that’s heavily entangled with ideas that are dangerous to spread? It’s true that there are dangerous incentive problems here, where basically people can unaccountably lie about their private insight into dangerous issues; on the other hand, it seems bad for ideas to be shared that are more or less plausible precursors to a world-ending artifact. My understanding about Eliezer and MIRI is basically, Eliezer wrote a bunch of public stuff that demonstrated that he has insight into the alignment problem, and professed his intent to solve alignment, and then he more or less got tenure from EA. Is that not what happened? Is that not what should have happened? That seems like the next best thing to directly sharing dangerous stuff.
I could imagine a lot of points of disagreement, like
1. that there’s such a thing as ideas that are plausible precursors to world-ending artifacts;
2. that some people should be funded to work on dangerous ideas that can’t be directly shared / evidenced;
3. that Eliezer’s public writing is enough to deserve “tenure”;
4. that the danger of sharing ideas that catalyze world-ending outweighs the benefits of understanding the alignment problem better and generally coordinating by sharing more.
The issue of people deciding to keep secrets is a separate issue from how *other people* should treat these “sorcerers”. My guess is that it’d be much better if sorcerers could be granted tenure without people trusting their opinions or taking instructions from them, when those opinions and instructions are based on work that isn’t shared. (This doesn’t easily mesh with intuitions about status: if someone should be given sorcerer tenure, isn’t that the same thing as them being generally trusted? But no, it’s not, it should be perfectly reasonable to believe someone is a good bet to do well within their cabal, but not a good bet to do well in a system that takes commands and deductions from hidden beliefs without sharing the hidden beliefs.)
Some ways of giving third parties Bayesian evidence that you have some secret without revealing it:
Demos, show off the capability somehow
Have the idea evaluated by a third party who doesn’t share it with the public
Do public work that is impressive the way you’re claiming the secret is (so it’s a closer analogy)
I’m not against “tenure” in this case. I don’t think it makes sense for people to make their plans around the idea that person X has secret Y unless they have particular reason to think secret Y is really important and likely to be possessed by person X (which is related to what you’re saying about trusting opinions and taking instructions). In particular, outsiders should think there’s ~0 chance that a particular AI researcher’s secrets are important enough here to be likely to produce AGI without some sort of evidence. Lots of people in the AI field say they have these sorts of secrets and many have somewhat impressive AI related accomplishments, they’re just way less impressive than what would be needed for outsiders to assign a non-negligible chance to possession of enough secrets to make AGI, given base rates.
The following is an edited partial chat transcript of a conversation involving me, Benquo, and an anonymous person (X). I am posting it in the hope that it has enough positive value-of-information compared to the attentional cost to be of benefit to others. I hope people can take this as “yay, I got to witness a back-room conversation” rather than “oh no, someone has made a bunch of public assertions that they can’t back up”; I think it would be difficult and time-consuming to argue for all these points convincingly, although I can explain to some degree, and I’m not opposed to people stating their disagreements. There were concerns about reputational risk as a result of posting this, which is partially mitigated by redaction, and partially is a cost that is intentionally “eaten” to gain the value-of-information benefit.
I began reading this charitably (unaware of whatever inside baseball is potentially going on, and seems to be alluded to), but to be honest struggled after “X” seemed to really want someone (Eliezer) to admit they’re “not smart”? I’m not sure why that would be relevant.
I think I found these lines especially confusing, if you want to explain:
“I just hope that people can generalize from “alignment is hard” to “generalized AI capabilities are hard”.
Is capability supposed to be hard for similar reasons as alignment? Can you expand/link? The only argument I can think of relating the two (which I think is a bad one) is “machines will have to solve their own alignment problem to become capable.”
Eliezer is invalidating the second part of this but not the first.
This would be a pretty useless machiavellian strategy, so I’m assuming you’re saying it’s happening for other reasons? Maybe self-deception? Can you explain?
Eliezer thinks that OpenAI will try to make things go faster rather than slower, but this is plainly inconsistent with things like the state of vitamin D research
This just made me go “wha” at first but my guess now is that this and the bits above it around speech recognition seem to be pointing at some AI winter-esque (or even tech stagnation) beliefs? Is this right?
I’m not sure exactly what is meant, one guess is that it’s about centrality: making yourself more central (more making executive decisions, more of a bottleneck on approving things, more looked to as a leader by others, etc) makes more sense the more you’re more correct about relevant things relative to other people. Saying “oh, I was wrong about a lot, whoops” is the kind of thing someone might do before e.g. stepping down as project manager or CEO. If you think your philosophy has major problems and your replacements’ philosophies have fewer major problems, that might increase the chance of success.
I would guess this is comparable to what Eliezer is saying in this post about how some people should just avoid consequentialist reasoning because they’re too bad at it and unlikely to improve:
...
Alignment is hard because it’s a quite general technical problem. You don’t just need to make the AI aligned in case X, you also have to make it aligned in cases Y and Z. To do this you need to create very general analysis and engineering tools that generalize across these situations.
Similarly, AGI is a quite general technical problem. You don’t just need to make an AI that can do narrow task X, it has to work in cases Y and Z too, or it will fall over and fail to take over the world at some point. To do this you need to create very general analysis and engineering tools that generalize across these situations.
For an intuition pump about this, imagine that LW’s effort towards making an aligned AI over the past ~14 years was instead directed at making AGI. We have records of certain mathematical formalisms people have come up with (e.g. UDT, logical induction). These tools are pretty far from enhancing AI capabilities. If the goal had been to enhance AI capabilities, they would have enhanced AI capabilities more, but still, the total amount of intellectual work that’s been completed is quite small compared to how much intellectual work would be required to build a working agent that generalizes across situations. The AI field has been at this for decades and has produced the results that it has, which are quite impressive in some domains, but still fail to generalize most of the time, and even what has been produced has required a lot of intellectual work spanning multiple academic fields and industries over decades. (Even if the field is inefficient in some ways, that would still imply that inefficiency is common, and LW seems to also be inefficient at solving AI-related technical problems compared to its potential.)
I’m not locating all the intentionality for creating these bubbles in Eliezer, there are other people in the “scene” that promote memes and gain various benefits from them (see this dialogue, ctrl-f “billions”).
There’s a common motive to try to be important by claiming that one has unique skills to solve important problems, and pursuing that motive leads to stress because it involves creating implicit or explicit promises that are hard to fulfil (see e.g. Elizabeth Holmes), and telling people “hey actually, I can’t solve this” reduces the stress level and makes it easier to live a normal life.
I think what Ben means here is that access to large amounts of capital is anti-correlated with actually trying to solve difficult intellectual problems. This is the opposite of what would be predicted by the efficient market hypothesis.
The Debtors’ Revolt argues that college (which many, many more Americans have gone too than previously) primarily functions to cause people to correlate with each other, not to teach people epistemic and instrumental rationality. E.g. college educated people are more likely to immediately dismiss Vitamin D as a COVID health intervention (due to an impression of “expert consensus”) rather than forming an opinion based on reading some studies and doing probability calculations. One would by default expect epistemic/instrumental rationality to be well-correlated with income, for standard efficient market hypothesis reasons. However, if there is a massive amount of correlation among the “irrational” actors, they can reward each other, provide insurance to each other, commit violence in favor of their class (e.g. the 2008 bailouts), etc.
(On this model, a major reason large companies do the “train a single large, expensive model using standard techniques like transformers” is to create correlation in the form of a canonical way of spending resources to advance AI.)
I don’t think this is a valid argument. Counter-example: you could build an AGI by uploading a human brain onto an artificial substrate, and you don’t “need to create very general analysis and engineering tools that generalize across these situations” to do this.
More realistically, it seems pretty plausible that all of the necessary patterns/rules/heuristics/algorithms/forms of reasoning necessary for “being generally intelligent” can be found in human culture, and ML can distill these elements of general intelligence into a (language or multimodal) model that will then be generally intelligent. This also doesn’t seem to require very general analysis and engineering tools. What do you think of this possibility?
You’re right that the uploading case wouldn’t necessarily require strong algorithmic insight. However, it’s a kind of bounded technical problem that’s relatively easy to evaluate progress in relative to the difficulty, e.g. based on ability to upload smaller animal brains, so would lead to >40 year timelines absent large shifts in the field or large drivers of progress. It would also lead to a significant degree of alignment by default.
For copying culture, I think the main issue is that culture is a protocol that runs on human brains, not on computers. Analogously, there are Internet protocols saying things like “a SYN/ACK packet must follow a SYN packet”, but these are insufficient for understanding a human’s usage of the Internet. Copying these would lead to imitations, e.g. machines that correctly send SYN/ACK packets and produce semi-grammatical text but lack certain forms of understanding, especially connection to a surrounding “the real world” that is spaciotemporal etc.
If you don’t have logic yourself, you can look at a lot of logical content (e.g. math papers) without understanding logic. Most machines work by already working, not by searching over machine designs that fit a dataset.
Also in the cultural case, if it worked it would be decently aligned, since it could copy cultural reasoning about goodness. (The main reason I have for thinking cultural notions of goodness might be undesirable is thinking that, as stated above, culture is just a protocol and most of the relevant value processing happens in the brains, see this post.)
Thanks so much for the one-paragraph summary of The Debtors’ Revolt, that was clarifying.
Glad this is shared.
There’s an awkward issue here, which is: how can there be people who are financially supported to do research on stuff that’s heavily entangled with ideas that are dangerous to spread? It’s true that there are dangerous incentive problems here, where basically people can unaccountably lie about their private insight into dangerous issues; on the other hand, it seems bad for ideas to be shared that are more or less plausible precursors to a world-ending artifact. My understanding about Eliezer and MIRI is basically, Eliezer wrote a bunch of public stuff that demonstrated that he has insight into the alignment problem, and professed his intent to solve alignment, and then he more or less got tenure from EA. Is that not what happened? Is that not what should have happened? That seems like the next best thing to directly sharing dangerous stuff.
I could imagine a lot of points of disagreement, like
1. that there’s such a thing as ideas that are plausible precursors to world-ending artifacts;
2. that some people should be funded to work on dangerous ideas that can’t be directly shared / evidenced;
3. that Eliezer’s public writing is enough to deserve “tenure”;
4. that the danger of sharing ideas that catalyze world-ending outweighs the benefits of understanding the alignment problem better and generally coordinating by sharing more.
The issue of people deciding to keep secrets is a separate issue from how *other people* should treat these “sorcerers”. My guess is that it’d be much better if sorcerers could be granted tenure without people trusting their opinions or taking instructions from them, when those opinions and instructions are based on work that isn’t shared. (This doesn’t easily mesh with intuitions about status: if someone should be given sorcerer tenure, isn’t that the same thing as them being generally trusted? But no, it’s not, it should be perfectly reasonable to believe someone is a good bet to do well within their cabal, but not a good bet to do well in a system that takes commands and deductions from hidden beliefs without sharing the hidden beliefs.)
Some ways of giving third parties Bayesian evidence that you have some secret without revealing it:
Demos, show off the capability somehow
Have the idea evaluated by a third party who doesn’t share it with the public
Do public work that is impressive the way you’re claiming the secret is (so it’s a closer analogy)
I’m not against “tenure” in this case. I don’t think it makes sense for people to make their plans around the idea that person X has secret Y unless they have particular reason to think secret Y is really important and likely to be possessed by person X (which is related to what you’re saying about trusting opinions and taking instructions). In particular, outsiders should think there’s ~0 chance that a particular AI researcher’s secrets are important enough here to be likely to produce AGI without some sort of evidence. Lots of people in the AI field say they have these sorts of secrets and many have somewhat impressive AI related accomplishments, they’re just way less impressive than what would be needed for outsiders to assign a non-negligible chance to possession of enough secrets to make AGI, given base rates.