Thanks for you feedback. I certainly appreciate your articles and I share many of your views. Reading what you had to say, along with Quentin, Jacob Cannell, Nora was a very welcome alternative take that expanded my thinking and changed my mind. I have changed my mind a lot over the last year, from thinking AI was a long way off and Yud/Bostrom were basically right to seeing that its a lot closer and theories without data are almost always wrong in may ways—e.g. SUSY was expected to be true for decades by most of the world’s smartest physicists. Many alignment ideas before GPT3.5 are either sufficiently wrong or irrelevant to do more harm than good.
Especially I think the over dependence on analogy, evolution. Sure when we had nothing to go on it was a start, but when data comes in, ideas based on analogies should be gone pretty fast if they disagree with hard data.
(Some background—I read the site for over 10 years have followed AI for my entire career, have an understanding of Maths, Psychology, and have built and deployed a very small NN model commercially. Also as an aside I remember distinctly being surprised that Yud was skeptical of NN/DL in the earlier days when I considered it obviously where AI progress would come from—I don’t have references because I didn’t think that would be disputed afterwards)
I am not sure what the silent majority belief on this site is (by people not Karma)? Is Yud’s worldview basically right or wrong?
Well they definitely can be applied there—though perhaps its a stage further than analogy and direct application of theory? Then of course data can agree/disagree.
gradient descent is not evolution and does not behave like evolution. it may still have problems one can imagine evolution having, but you can’t assume facts about evolution generalize—it’s in fact quite different.
e.g. SUSY was expected to be true for decades by most of the world’s smartest physicists.
I really don’t want to go down a rabbit hole here, so probably won’t engage in further discussion, but I just want to chime in here and say that I’m pretty sure lots of the world’s smartest physicists (not sure what fraction) still expect the fundamental laws of physics in our universe to have (broken) supersymmetry, and I would go further and say that they have numerous very good reasons to expect that, like gauge coupling unification etc. Same as ever. The fact that supersymmetric partners were not found at LHC is nonzero evidence against supersymmetric partners existing, but it’s not strong evidence against them existing, because LHC was very very far from searching the whole space of possibilities. Also, we pretty much know for a fact that the universe contains at least one other yet-to-be-discovered elementary particle beyond the 17 (or whatever, depends on how you count) particles in the Standard Model. So I think it’s extremely premature to imply that the prediction of yet-to-be-discovered supersymmetric partner particles has been ruled out in our universe and haha look at those overconfident theoretical physicists. (A number of specific SUSY-involving theories have been ruled out, but I think the smart physicists knew all along that those were just plausible hypotheses worth checking, not confident theoretical predictions.)
OK you are answering at a level more detailed than I raised and seem to assume I didn’t consider such things. My reason and IMO the expected reading of “SUSY has failed” is not that such particles have been ruled out as I know they havn’t, but that its theoretical benefits are severely weakened or entirely ruled out according to recent data. My reference to SUSY was specifically regarding its opportunity to solve the Hierarchy Problem. This is the common understanding of one of the reasons it was proposed.
I stand by my claim that many/most of the top physicists expected for >1 decade that it would help solve such a problem. I disagree with the claim:
“but I think the smart physicists knew all along that those were just plausible hypotheses worth checking, ” Smart physicists thought SUSY would solve the hierarchy problem.
----
Common knowledge, from GPT4:
“can SUSY still solve the Hierarchy problem with respect to recent results”
Hierarchy Problem: SUSY has been considered a leading solution to the hierarchy problem because it naturally cancels out the large quantum corrections that would drive the Higgs boson mass to a very high value. However, the non-observation of supersymmetric particles at expected energy levels has led some physicists to question whether SUSY can solve the hierarchy problem in its simplest forms.
Fine-Tuning: The absence of low-energy supersymmetry implies a need for fine-tuning in the theory, which contradicts one of the primary motivations for SUSY as a solution to the hierarchy problem. This has led to exploration of more complex SUSY models, such as those with split or high-scale supersymmetry, where SUSY particles exist at much higher energy scales.
----
IMO ever more complex models rapidly become like epi-cycles.
I am not sure what the silent majority belief on this site is (by people not Karma)? Is Yud’s worldview basically right or wrong?
I think this will depend strongly on where you draw the line on “basically”. I think the majority probably thinks:
AI is likely to be a really big deal
Existential risk from AI is at least substantial (e.g. >5%)
AI takeoff is reasonably likely to happen quite quickly in wall clock time if this isn’t actively prevented (e.g. AI will cause there to be <10 years from a 20% annualized GDP growth rate to a 100x annualized growth rate)
The power of full technological maturity is extremely high (e.g. nanotech, highly efficient computing, etc.)
But, I expect that the majority of people don’t think:
Inside view, existential risk is >95%
A century of dedicated research on alignment (targeted as well as society would realistically do) is insufficient to get risk <15%.
Yes to AI being a big deal and extremely powerful ( yes I doubt anyone would be here otherwise)
Yes—Don’t think anyone can reasonably claim its <5% but then so is not having AI if x-risk is defined to be humanity missing practically all of its Cosmic endowment.
Maybe—Even with slow takeoff, and hardware constrained you get much greater GDP, though I don’t agree with 100x (for the critical period that is, 100x could happen later). E.g. car factories are made to produce robots, we get 1-10 billion more minds and bodies per year, but not quite 100X. ~10x per year is enough to be extremely disruptive and x-risk anyway.
---
(1)
Yes I don’t think x-risk is >95% - say 20% as a very rough guess that humanity misses all its Cosmic endowment. I think AI x-risk needs to be put in this context—say you ask someone
“What’s the chance that humanity becomes successfully interstellar?”
If they say 50⁄50 then being OK with any AI x-risk less than 50% is quite defensible if getting AI right means that its practically certain you get your cosmic endowment etc.
---
(2)
I do think its defensible that a century of dedicated research on alignment doesn’t get risk <15% but because alignment research is only useful a little bit in advance of capabilities—say we had a 100 year pause, then I wouldn’t have confidence in our alignment plan at the end of it.
Anyway regarding x-risk I don’t think there is a completely safe path. Too fast with AI and obvious risk, too slow and there is also other obvious risks. Our current situation is likely unstable. For example the famous quote
“If you want a picture of the future, imagine a boot stamping on a human face— forever.”
I believe that is now possible with current tech, where it was not say for Soviet Russia. So we may be in the situation where societies can go 1984 totalitarian bad, but not come back because our tech coordination skills are sufficient to stop centralized empires from collapsing. LLM of course make censorship even easier. (I am sure there are other ways our current tech could destroy most societies also)
If that’s the case, a long pause could result in all power being in such societies which when the pause ended would be very likely to screw up alignment.
That makes me unsure what regulation to advocate for, though I am in favor of slowing down hardware AI progress but fully exploring the capabilities of our current HW.
Most importantly I think we should hugely speed up Neuralink type devices and brain uploading. I would identify much more with an uploaded human that was then carefully, appropriately upgraded to superintelligence than an alternative path where a pure AI superintelligence was made.
We have to accept that we live in critical times and just slowing things down is not necessarily the safest option.
Thanks for you feedback. I certainly appreciate your articles and I share many of your views. Reading what you had to say, along with Quentin, Jacob Cannell, Nora was a very welcome alternative take that expanded my thinking and changed my mind. I have changed my mind a lot over the last year, from thinking AI was a long way off and Yud/Bostrom were basically right to seeing that its a lot closer and theories without data are almost always wrong in may ways—e.g. SUSY was expected to be true for decades by most of the world’s smartest physicists. Many alignment ideas before GPT3.5 are either sufficiently wrong or irrelevant to do more harm than good.
Especially I think the over dependence on analogy, evolution. Sure when we had nothing to go on it was a start, but when data comes in, ideas based on analogies should be gone pretty fast if they disagree with hard data.
(Some background—I read the site for over 10 years have followed AI for my entire career, have an understanding of Maths, Psychology, and have built and deployed a very small NN model commercially. Also as an aside I remember distinctly being surprised that Yud was skeptical of NN/DL in the earlier days when I considered it obviously where AI progress would come from—I don’t have references because I didn’t think that would be disputed afterwards)
I am not sure what the silent majority belief on this site is (by people not Karma)? Is Yud’s worldview basically right or wrong?
analogies based on evolution should be applied at the evolutionary scale: between competing organizations.
Well they definitely can be applied there—though perhaps its a stage further than analogy and direct application of theory? Then of course data can agree/disagree.
gradient descent is not evolution and does not behave like evolution. it may still have problems one can imagine evolution having, but you can’t assume facts about evolution generalize—it’s in fact quite different.
I really don’t want to go down a rabbit hole here, so probably won’t engage in further discussion, but I just want to chime in here and say that I’m pretty sure lots of the world’s smartest physicists (not sure what fraction) still expect the fundamental laws of physics in our universe to have (broken) supersymmetry, and I would go further and say that they have numerous very good reasons to expect that, like gauge coupling unification etc. Same as ever. The fact that supersymmetric partners were not found at LHC is nonzero evidence against supersymmetric partners existing, but it’s not strong evidence against them existing, because LHC was very very far from searching the whole space of possibilities. Also, we pretty much know for a fact that the universe contains at least one other yet-to-be-discovered elementary particle beyond the 17 (or whatever, depends on how you count) particles in the Standard Model. So I think it’s extremely premature to imply that the prediction of yet-to-be-discovered supersymmetric partner particles has been ruled out in our universe and haha look at those overconfident theoretical physicists. (A number of specific SUSY-involving theories have been ruled out, but I think the smart physicists knew all along that those were just plausible hypotheses worth checking, not confident theoretical predictions.)
OK you are answering at a level more detailed than I raised and seem to assume I didn’t consider such things. My reason and IMO the expected reading of “SUSY has failed” is not that such particles have been ruled out as I know they havn’t, but that its theoretical benefits are severely weakened or entirely ruled out according to recent data. My reference to SUSY was specifically regarding its opportunity to solve the Hierarchy Problem. This is the common understanding of one of the reasons it was proposed.
I stand by my claim that many/most of the top physicists expected for >1 decade that it would help solve such a problem. I disagree with the claim:
“but I think the smart physicists knew all along that those were just plausible hypotheses worth checking, ” Smart physicists thought SUSY would solve the hierarchy problem.
----
Common knowledge, from GPT4:
“can SUSY still solve the Hierarchy problem with respect to recent results”
Hierarchy Problem: SUSY has been considered a leading solution to the hierarchy problem because it naturally cancels out the large quantum corrections that would drive the Higgs boson mass to a very high value. However, the non-observation of supersymmetric particles at expected energy levels has led some physicists to question whether SUSY can solve the hierarchy problem in its simplest forms.
Fine-Tuning: The absence of low-energy supersymmetry implies a need for fine-tuning in the theory, which contradicts one of the primary motivations for SUSY as a solution to the hierarchy problem. This has led to exploration of more complex SUSY models, such as those with split or high-scale supersymmetry, where SUSY particles exist at much higher energy scales.
----
IMO ever more complex models rapidly become like epi-cycles.
I think this will depend strongly on where you draw the line on “basically”. I think the majority probably thinks:
AI is likely to be a really big deal
Existential risk from AI is at least substantial (e.g. >5%)
AI takeoff is reasonably likely to happen quite quickly in wall clock time if this isn’t actively prevented (e.g. AI will cause there to be <10 years from a 20% annualized GDP growth rate to a 100x annualized growth rate)
The power of full technological maturity is extremely high (e.g. nanotech, highly efficient computing, etc.)
But, I expect that the majority of people don’t think:
Inside view, existential risk is >95%
A century of dedicated research on alignment (targeted as well as society would realistically do) is insufficient to get risk <15%.
Which I think are both beliefs Yudkowsky has.
For me -
Yes to AI being a big deal and extremely powerful ( yes I doubt anyone would be here otherwise)
Yes—Don’t think anyone can reasonably claim its <5% but then so is not having AI if x-risk is defined to be humanity missing practically all of its Cosmic endowment.
Maybe—Even with slow takeoff, and hardware constrained you get much greater GDP, though I don’t agree with 100x (for the critical period that is, 100x could happen later). E.g. car factories are made to produce robots, we get 1-10 billion more minds and bodies per year, but not quite 100X. ~10x per year is enough to be extremely disruptive and x-risk anyway.
---
(1)
Yes I don’t think x-risk is >95% - say 20% as a very rough guess that humanity misses all its Cosmic endowment. I think AI x-risk needs to be put in this context—say you ask someone
“What’s the chance that humanity becomes successfully interstellar?”
If they say 50⁄50 then being OK with any AI x-risk less than 50% is quite defensible if getting AI right means that its practically certain you get your cosmic endowment etc.
---
(2)
I do think its defensible that a century of dedicated research on alignment doesn’t get risk <15% but because alignment research is only useful a little bit in advance of capabilities—say we had a 100 year pause, then I wouldn’t have confidence in our alignment plan at the end of it.
Anyway regarding x-risk I don’t think there is a completely safe path. Too fast with AI and obvious risk, too slow and there is also other obvious risks. Our current situation is likely unstable. For example the famous quote
“If you want a picture of the future, imagine a boot stamping on a human face— forever.”
I believe that is now possible with current tech, where it was not say for Soviet Russia. So we may be in the situation where societies can go 1984 totalitarian bad, but not come back because our tech coordination skills are sufficient to stop centralized empires from collapsing. LLM of course make censorship even easier. (I am sure there are other ways our current tech could destroy most societies also)
If that’s the case, a long pause could result in all power being in such societies which when the pause ended would be very likely to screw up alignment.
That makes me unsure what regulation to advocate for, though I am in favor of slowing down hardware AI progress but fully exploring the capabilities of our current HW.
Most importantly I think we should hugely speed up Neuralink type devices and brain uploading. I would identify much more with an uploaded human that was then carefully, appropriately upgraded to superintelligence than an alternative path where a pure AI superintelligence was made.
We have to accept that we live in critical times and just slowing things down is not necessarily the safest option.
Yup, and this is why I’m more excited to supervise MATS mentees who haven’t read The Sequences.