But in fact there’s no one base rate; instead, different subagents with different domains of knowledge will have different base rates. That will push P(doom) lower because most frames from most disciplines, and most styles of reasoning, don’t predict doom. That’s where the asymmetry which makes 90% a much stronger prediction than 10% comes from.
One of the most important features of future ASI I consider knowledge of limits of applicability of its models and heuristics. If you have list of assumptions for very fast heuristics, then you can win big by doing fast-computable moves in narrow environment where assumptions hold. Thus saying, you need to be able find when your assumptions don’t hold and command your subagents to halt, melt and catch fire when they are outside of their applicability zone.
One of the most important features of future ASI I consider knowledge of limits of applicability of its models and heuristics. If you have list of assumptions for very fast heuristics, then you can win big by doing fast-computable moves in narrow environment where assumptions hold. Thus saying, you need to be able find when your assumptions don’t hold and command your subagents to halt, melt and catch fire when they are outside of their applicability zone.