That seems like extremely limited, human thinking. If we’re assuming a super powerful AGI, capable of wiping out humanity with high likelihood, it is also almost certainly capable of accomplishing its goals despite our theoretical attempts to stop it without needing to kill humans. The issue, then, is not fully aligning AGI goals with human goals, but ensuring it has “don’t wipe out humanity, don’t cause extreme negative impacts to humanity” somewhere in its utility function. Probably doesn’t even need to be weighted too strongly, if we’re talking about a truly powerful AGI. Chimpanzees presumably don’t want humans to rule the world—yet they have made no coherent effort to stop us from doing so, probably haven’t even realized we are doing so, and even if they did we could pretty easily ignore it.
“If something could get in the way (or even wants to get in the way, whether or not it is capable of trying) I need to wipe it out” is a sad, small mindset and I am entirely unconvinced that a significant portion of hypothetically likely AGIs would think this way. I think AGI will radically change the world, and maybe not for the better, but extinction seems like a hugely unlikely outcome.
Other people have addressed the truth/belief gap. I want to talk about existential risk.
We got EXTREMELY close to extinction with nukes, more than once. Launch orders in the Cold War were given and ignored or overridden three separate times that I’m aware of, and probably more. That risk has declined but is still present. The experts were 100% correct and their urgency and doomsday predictions were arguably one of the reasons we are not all dead.
The same is true of global warming, and again there is still some risk. We probably got extremely lucky in the last decade and happened upon the right tech and strategies and got decent funding to combat climate change such that it won’t reach 3+ degrees deviation, but that’s still not a guarantee and it also doesn’t mean the experts were wrong. It was an emergency, it still is, the fact that we got lucky doesn’t mean we shouldn’t have paid very close attention.
The fact that we might survive this potential apocalypse too is not a reason to act like it is not a potential apocalypse. I agree that empirically, humans have a decent record at avoiding extinction when a large number of scientific experts predict its likelihood. It’s not a great record, we’re like 4-0 depending on how you count, which is not many data points, but it’s something. What we have learned from those experiences is that the loud and extreme actions of a small group of people who are fully convinced of the risk is sometimes enough to sufficiently shift the inertia of a large society only vaguely aware of the risk to avoid catastrophe by a hairs breadth. We might need to be that group.