Bengio and Hinton are the two most influential “old guard” AI researchers turned safety advocates as far as I can tell, with Bengio being more active in research. Your e.g. is super misleading, since my list would have been something like:
Bengio
Hinton
Russell
One has to be a bit careful with this though. E.g. someone experiencing or having experienced harassment may have a seemingly pathological obsession on the circumstances and people involved in the situation, but it may be completely proportional to the way that it affected them—it only seems pathological to people who didn’t encounter the same issues.
I think it’s not an unreasonable point to take into account when talking price, but also a lot of the time it’s serves as a BS talking point for people who don’t really care about the subtleties.
I’m definitely fine with not having Superman, but I’m willing to settle on him not intervening.
On a different note, I’d disagree that Superman, just by existing and being powerful, is a de facto ruler in any sense—he of course could be, but that would entail a tradeoff that he may not like (living an unburdened life).
In what way do prediction markets provide significant evidence on this type of question?
He’s clearly not completely discounting that there’s progress, but overall it doesn’t feel like he’s “updating all the way”:
This is a recent post about the deepmind math olympiad results: https://mathstodon.xyz/@tao/112850716240504978
“1. This is great work, shifting once again our expectations of which benchmark challenges are within reach of either #AI-assisted or fully autonomous methods”
Thanks for explaining your point—that viability of inference scaling makes development differentially safer (all else equal) seems right.
Tall buildings are very predictable, and you can easily iterate on your experience before anything can really go wrong. Nuclear bombs is similar (you can in principle test in a remote enough location).
Biological weapons seems inherently more dangerous (still overall more predictable than AI), and I’d naively imagine it to be simply very risky to develop extremely potent biological weapons.
Interestingly, Terence Tao has recently started thinking about AI, and his (publicly stated) opinions on it are … very conservative? I find he mostly focuses on the capabilities that are already here and doesn’t really extrapolate from it in any significant way.
I think you’re getting this exactly wrong (and this invalidates most of the OP). If you find a model that has a constant factor of 100 in the asymptotics that’s a huge deal if everything else has log scaling. That would already present discontinuous progress and potentially put you at ASI right away.
Basically the current scaling laws, if they keep holding are a lower bound on the expected progress and can’t really give you any information to upper bound it.
It’s certainly covered by the NYT, although their angle is “OpenAI is growing up”.
Some reporting on this: https://www.vox.com/future-perfect/374275/openai-just-sold-you-out
I understand that this hit a bad tone, but I do kind of stand behind the original comment for the precise reason that Altman has been particularly good at weaponizing things like “fun” and “high status” which the OP plays right into.
Bernard Arnault?
I don’t have any experience with actual situations where this could be relevant, but it does feel like you’re overly focusing on the failure case where everyone is borderline incompetent and doing arbitrary things (which of course happens on less wrong sometimes, since the variation here is quite large!). There’s clearly a huge upside to being able to spot when you’re trying to do something that’s impossible for theoretical reasons, and being extra sceptical in these situations. (E.g. someone trying to construct a perpetual motion machine). I’m open to the argument that there’s a lot to be wished for in the way people in practice apply these things.
(Epistemic status: I know basic probability theory but am otherwise just applying common sense here)
This seems to mostly be a philosophical question. I believe the answer is that then you’re hitting the limits of your model and Bayesianism doesn’t necessarily apply. In practical terms, I’d say it’s most likely that you were mistaken about the probability of the event in fact being 0. (Probability 1 events occuring should be fine).
In 2021, I predicted math to be basically solved by 2023 (using the kind of reinforcement learning on formally checkable proofs that deepmind is using). It’s been slower than expected and I wouldn’t have guessed some less formal setting like o1 to go relatively well—but since then I just nod along to these kinds of results.
(Not sure what to think of that claimed 95% number though—wouldn’t that kind of imply they’d blown past the IMO grand challenge? EDIT: There were significant time limits on the human participants, see Qumeric’s comment.)
I think the main problem is that society-at-large doesn’t significantly value AI safety research, and hence that the funding is severely constrained. I’d be surprised if the consideration you describe in the last paragraph plays a significant role.
Ideas come from unsupervised training, answers from supervised training and proofs from RL on a specified reward function.
Is there a reason to be so specific, or could one equally well formulate this more generally?