Fundamentally, OP is making the case that Biorisk is an extremely helpful (but not exact) analogy for AI risk, in the sense that we can gain understanding by looking the ways it’s analogous, and then get an even more nuanced understanding by analyzing the differences.
The point made seems to be more about it’s place in the discourse than about the value of the analogy itself? (E.g. “The Biorisk analogy is over-used” would be less misleading then)
What do you think about building legal/technological infrastructure to enable a prompt pause, should it seem necessary?
I agree that potentially the benefits can go to everyone. The point is that as the person pursuing AGI you are making the choice for everyone else.
The asymmetry is that if you do something that creates risk for everyone else, I believe that does single you out as an aggressor? While conversely, enforcing norms that prevent such risky behavior seems justified. The fact that by default people are mortal is tragic, but doesn’t have much bearing here. (You’d still be free to pursue life-extension technology in other ways, perhaps including limited AI tools).
Ideally, of course, there’d be some sort of democratic process here that let’s people in aggregate make informed (!) choices. In the real world, it’s unclear what a good solution here would be. What we have right now is the big labs creating facts that society has trouble catching up with, which I think many people are reasonably uncomfortable with.
I think the perspective that you’re missing regarding 2. is that by building AGI one is taking the chance of non-consensually killing vast amounts of people and their children for some chance of improving one’s own longevity.
Even if one thinks it’s a better deal for them, a key point is that you are making the decision for them by unilaterally building AGI. So in that sense it is quite reasonable to see it as an “evil” action to work towards that outcome.
Somewhat of a nitpick, but the relevant number would be p(doom | strong AGI being built) (maybe contrasted with p(utopia | strong AGI)) , not overall p(doom).
I was down voting this particular post because I perceived it as mostly ideological and making few arguments, only stating strongly that government action will be bad. I found the author’s replies in the comments much more nuanced and would not have down-voted if I’d perceived the original post to be of the same quality.
Basically, I think whether or not one thinks whether alignment is hard or not is much more of the crux than whether or not they’re utilitarian.
Pesonally, I don’t find Pope & Belrose very convincing, although I do commend them for the reasonable effort—but if I did believe that AI is likely to go well, I’d probably also be all for it. I just don’t see how this is related to utilitarianism (maybe for all but a very small subset of people in EA).
“One reason that goes overlooked is that most human beings are not utilitarians” I think this point is just straightforwardly wrong. Even from a purely selfish perspective, it’s reasonable to want to stop AI.
The main reason humanity is not going to stop seems mainly like coordination problems, or something close to learned helplessness in these kind of competitive dynamics.
Unambiguously evil seems unnecessarily strong. Something like “almost certainly misguided” might be more appropriate? (still strong, but arguably defensible)
Do you have an example for where better conversations are happening?
You’re conflating between “have important consequences” and “can be used as weapons in discourse”
What do you mean by example, here? That this is demonstrating a broader property, or that in this situation, there was a tribal dynamic?
Typo, thanks for pointing it out. Also, see here for the physics reference: https://en.m.wikipedia.org/wiki/1_%2B_2_%2B_3_%2B_4_%2B_⋯
I’m one of these professional mathematicians, and I’ll say that this article completely fails to demonstrate it’s central thesis that there is a valid intuitive argument for concluding that 1 + 2 + 3 + … = −1/12 makes sense. What’s worse, it only pretends to do so by what’s essentially a swindle. In my understanding, it’s relatively easy to reason that a given divergent series “should” take an arbitrary finite value by the kind of arguments employed here, so what is being done is taking a foregone conclusion and providing some false intuition for why it should be true.
On a less serious note, speaking to the real reason why 1 + 2 + 3 + … = −1/12, that’s actually what physicists will tell you, and we all know one should be careful around those.
Sure, but you’re just claiming that, and I don’t think it’s actually true.
You run into the trouble of having to defend why your way to fit the divergent series into a pattern is the right one—other approaches may give different results.
I think it’s quite unlikely that GPT 5 will destroy the world. That said, I think it’s generally reasonable to doubt prediction markets on questions that can’t be fairly evaluated both ways.
I think the possibility of compute overhang seems plausible given the technological realities, but generalizing from this to a second-order overhang etc. seems taking it too far.
If there is an argument that we should push compute due to the danger of another “overhang” down the line that should be made explicitly and not by generalisation from one (debatable!) example.
Sorry, low effort comment on my side. Still, I think the original link seems misleading in the point it’s purportedly trying to make.
Half a year ago, I’d have guessed that OpenAI leadership, while likely misguided, was essentially well-meaning and driven by a genuine desire to confront a difficult situation. The recent series of events has made me update significantly against the general trustworthiness and general epistemic reliability of Altman and his circle. While my overall view of OpenAI’s strategy hasn’t really changed, my likelihood of them possibly “knowing better” has dramatically gone down now.