Figure 1 in Carroll’s paper shows what is going on. At the base is the fundamental “Underlying reality” which we don’t yet understand (eg. it might be string theory or cellular automata, etc.):
Above that is the “Quantum Field Theory” level which includes the “Core Theory” which he explicitly shows in the paper and also possibly “Unknown particles and forces”. Above that is the “Macro Level” which includes both “Everyday life” which he is focusing on and also “Astrophysics and Cosmology”. His claim is that everything we experience in the “Everyday life” level depends on the “Underlying reality” level only through the “Core Theory” (ie. it is an “effective theory” kind of like fluid mechanics doesn’t depend on the details of particle interactions).
In particular, for energies less than 10^11 electron volts and for gravitational fields weaker than those around black holes, neutron stars, and the early universe, the results of every experiment is predicted by the Core theory to very high accuracy. If anything in this regime were not predicted to high accuracy, it would be front page news, the biggest development in physics in 50 years, etc. Part of this confidence arises from fundamental aspects of physics: locality of interaction, conservation of mass/energy, and symmetry under the Poincare group. These have been validated in every experiment ever conducted. Of course, as you say, physics isn’t finished and quantum theory in high gravitational curvature is still not understood.
Here’s a list of other unsolved problems in physics: https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_physics But the key point is that none of these impact AI safety (at least in the nearterm!). Certainly, powerful adversarial AI will look for flaws in our model of the universe as a potential opportunity for exploitation. Fortunately, we have a very strong current theory and we can use it to put bounds on the time and energy an AI would require to violate the conditions of validity (eg. create black holes, etc.) For long term safety and stability, humanity will certainly have to put restrictions on those capabilities, at least until the underlying physics is fully understood.
In particular, for energies less than 10^11 electron volts and for gravitational fields weaker than those around black holes, neutron stars, and the early universe, the results of every experiment is predicted by the Core theory to very high accuracy. If anything in this regime were not predicted to high accuracy, it would be front page news, the biggest development in physics in 50 years, etc. Part of this confidence arises from fundamental aspects of physics: locality of interaction, conservation of mass/energy, and symmetry under the Poincare group. These have been validated in every experiment ever conducted. Of course, as you say, physics isn’t finished and quantum theory in high gravitational curvature is still not understood.
While I am an avid physics reader, I don’t have a degree in physics, so this is speaking at the level of an informed layman.
I think it’s actually pretty easy to end up with small concentrations of more than 10^11 electron volts and large local gravitational fields.These effects can then often ripple out or qualitatively change the character of some important interaction. On the everyday scale, cosmic rays are the classical example of extremely high-energy contexts, which do effect us on a daily level (but of course there are many more contexts in which local bubbles of high energy concentration takes place).
Also, dark energy + dark matter are of course the obvious examples of something for which we currently have no satisfying explanation within either general relativity or the standard model, and neither of those likely requires huge energy scales or large gravitational fields.
In general, I don’t think it’s at all true that “if anything was not predicted with high accuracy by the standard model it would be the biggest development in physics in 50 years”. We have no idea what the standard model predicts about approximately any everyday phenomena because simulating phenomena at the everyday scale is completely computationally intractable. If turbulence dynamics or common manufacturing or material science observations were in conflict with the standard model, we would have no idea, since we have no idea what the standard model says about basically any of those things.
In the history of science it’s quite common that you are only able to notice inconsistencies in your previous theory after you have found a superior theory. Newton’s gravity looks great for predicting the movements of the solar system, with a pretty small error that mostly looks random and you can probably just dismiss as measurement error, until you have relativity and you notice that there was a systematic bias in all of your measurements in predictable directions in a way that previously looked like noise.
It’s also quite challenging to create high energy particles, they tend to rapidly collide and dissipate their energy. The CERN “Large Hadron Collider” is the most powerful particle accelerator that humans have built: https://home.cern/resources/faqs/facts-and-figures-about-lhc It involves 27 kilometers of superconducting magnets and produces proton collisions of 1.3*10^13eV.
Most cosmic rays are in the range of 10^6 eV to 10^9 eV https://news.uchicago.edu/explainer/what-are-cosmic-rays But there have been a few very powerful cosmic rays detected. Betwen 2004 and 2007, the Pierre Auger Observatory detected 27 events with energies above 5.7 * 10^19 eV and the “Oh-My-God” particle detected in 1991 had an energy of 3.2 * 10^20 eV.
So they can happen but would be extremely difficult for an adversary to generate. The only reason he put 10^11 as a limit is that’s the highest we’ve been able to definitively explore with accelerators. There may be more unexpected particles up there, but I don’t think they would make much of a difference to the kinds of devices we’re talking about.
But we certainly have to be vigilant! ASIs will likely explore every avenue and may very well be able to discover the “Theory of Everything”. We need to design our systems so that we can update them with new knowledge. Ideally we would also have confidence that our infrastructure could detect attempts to subvert it by pushing outside the domain of validity of our models.
While dark energy and dark matter have a big effect on the evolution of the universe as a whole, they don’t interact in any measurable way with systems here on earth. Ethan Siegel has some great posts narrowing down their properties based on what we definitively know, eg. https://bigthink.com/starts-with-a-bang/dark-matter-bullet-cluster/ So it’s important on large scales but not, say, on the scale of earth. Of course, if we consider the evolution of AI and humanity over much longer timescales, then we will likely need a detailed theory. That again shows that we need to work with precise models which may expand their regimes of applicability.
Even if everything is in principle calculable, it doesn’t mean you can do useful calculations of complex systems a useful distance into the future. The three body problem intervenes. And there are rather more than three bodies if you’re trying to predict behavior of a brain-sized neural network, let alone intervening on a complex physical world. The computer you’d need wouldn’t just be the size of the universe, but all of the many worlds branches.
Simulation of the time evolution of models from their dynamical equations is only one way of proving properties about them. For example, a harmonic oscillator https://en.wikipedia.org/wiki/Harmonic_oscillator has dynamical equations m d^2x/dt^2= -kx. You can simulate that but you can also prove that the kinetic plus potential energy is conserved and get limits on its behavior arbitrarily far into the future.
Sure but seems highly unlikely there are any such neat simplifications for complex cognitive systems built from neural networks.
Other than “sapient beings do things that further their goals in their best estimation”, which is a rough predictor, and what we’re already trying to focus on. But the devil is in the details, and the important question is about how the goal is represented and understood.
Oh yeah, by their very nature it’s likely to be hard to predict intelligent systems behavior in detail. We can put constraints on them, though, and prove that they operate within those constraints.
Even simple systems like random SAT problems https://en.wikipedia.org/wiki/SAT_solver can have a very rich statistical structure. And the behavior of the solvers can be quite unpredictable.
On the other hand, I think it is often possible to distill behavior for a particlular task from a rich intelligence into simple code with provable properties.
Figure 1 in Carroll’s paper shows what is going on. At the base is the fundamental “Underlying reality” which we don’t yet understand (eg. it might be string theory or cellular automata, etc.):
Above that is the “Quantum Field Theory” level which includes the “Core Theory” which he explicitly shows in the paper and also possibly “Unknown particles and forces”. Above that is the “Macro Level” which includes both “Everyday life” which he is focusing on and also “Astrophysics and Cosmology”. His claim is that everything we experience in the “Everyday life” level depends on the “Underlying reality” level only through the “Core Theory” (ie. it is an “effective theory” kind of like fluid mechanics doesn’t depend on the details of particle interactions).
In particular, for energies less than 10^11 electron volts and for gravitational fields weaker than those around black holes, neutron stars, and the early universe, the results of every experiment is predicted by the Core theory to very high accuracy. If anything in this regime were not predicted to high accuracy, it would be front page news, the biggest development in physics in 50 years, etc. Part of this confidence arises from fundamental aspects of physics: locality of interaction, conservation of mass/energy, and symmetry under the Poincare group. These have been validated in every experiment ever conducted. Of course, as you say, physics isn’t finished and quantum theory in high gravitational curvature is still not understood.
Here’s a list of other unsolved problems in physics: https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_physics But the key point is that none of these impact AI safety (at least in the nearterm!). Certainly, powerful adversarial AI will look for flaws in our model of the universe as a potential opportunity for exploitation. Fortunately, we have a very strong current theory and we can use it to put bounds on the time and energy an AI would require to violate the conditions of validity (eg. create black holes, etc.) For long term safety and stability, humanity will certainly have to put restrictions on those capabilities, at least until the underlying physics is fully understood.
While I am an avid physics reader, I don’t have a degree in physics, so this is speaking at the level of an informed layman.
I think it’s actually pretty easy to end up with small concentrations of more than 10^11 electron volts and large local gravitational fields.These effects can then often ripple out or qualitatively change the character of some important interaction. On the everyday scale, cosmic rays are the classical example of extremely high-energy contexts, which do effect us on a daily level (but of course there are many more contexts in which local bubbles of high energy concentration takes place).
Also, dark energy + dark matter are of course the obvious examples of something for which we currently have no satisfying explanation within either general relativity or the standard model, and neither of those likely requires huge energy scales or large gravitational fields.
In general, I don’t think it’s at all true that “if anything was not predicted with high accuracy by the standard model it would be the biggest development in physics in 50 years”. We have no idea what the standard model predicts about approximately any everyday phenomena because simulating phenomena at the everyday scale is completely computationally intractable. If turbulence dynamics or common manufacturing or material science observations were in conflict with the standard model, we would have no idea, since we have no idea what the standard model says about basically any of those things.
In the history of science it’s quite common that you are only able to notice inconsistencies in your previous theory after you have found a superior theory. Newton’s gravity looks great for predicting the movements of the solar system, with a pretty small error that mostly looks random and you can probably just dismiss as measurement error, until you have relativity and you notice that there was a systematic bias in all of your measurements in predictable directions in a way that previously looked like noise.
It’s very hard to get large gravitational fields. The closest known black hole to Earth is Gaia BH1 which is 1560 light-years away: https://www.space.com/closest-massive-black-hole-earth-hubble The strongest gravitational waves come from the collision of two black holes but by the time they reach Earth they are so weak it takes huge effort to measure them and they are in the weak curvature regime where standard quantum field theory is fine: https://www.ligo.caltech.edu/page/what-are-gw#:~:text=The%20strongest%20gravitational%20waves%20are,)%2C%20and%20colliding%20neutron%20stars.
It’s also quite challenging to create high energy particles, they tend to rapidly collide and dissipate their energy. The CERN “Large Hadron Collider” is the most powerful particle accelerator that humans have built: https://home.cern/resources/faqs/facts-and-figures-about-lhc It involves 27 kilometers of superconducting magnets and produces proton collisions of 1.3*10^13eV.
Most cosmic rays are in the range of 10^6 eV to 10^9 eV https://news.uchicago.edu/explainer/what-are-cosmic-rays But there have been a few very powerful cosmic rays detected. Betwen 2004 and 2007, the Pierre Auger Observatory detected 27 events with energies above 5.7 * 10^19 eV and the “Oh-My-God” particle detected in 1991 had an energy of 3.2 * 10^20 eV.
So they can happen but would be extremely difficult for an adversary to generate. The only reason he put 10^11 as a limit is that’s the highest we’ve been able to definitively explore with accelerators. There may be more unexpected particles up there, but I don’t think they would make much of a difference to the kinds of devices we’re talking about.
But we certainly have to be vigilant! ASIs will likely explore every avenue and may very well be able to discover the “Theory of Everything”. We need to design our systems so that we can update them with new knowledge. Ideally we would also have confidence that our infrastructure could detect attempts to subvert it by pushing outside the domain of validity of our models.
While dark energy and dark matter have a big effect on the evolution of the universe as a whole, they don’t interact in any measurable way with systems here on earth. Ethan Siegel has some great posts narrowing down their properties based on what we definitively know, eg. https://bigthink.com/starts-with-a-bang/dark-matter-bullet-cluster/ So it’s important on large scales but not, say, on the scale of earth. Of course, if we consider the evolution of AI and humanity over much longer timescales, then we will likely need a detailed theory. That again shows that we need to work with precise models which may expand their regimes of applicability.
An example of this kind of thing is the “Proton Radius Puzzle” https://physicsworld.com/a/solving-the-proton-puzzle/ https://en.wikipedia.org/wiki/Proton_radius_puzzle in which different measurements and theoretical calculations of the radius of the proton differed by about 4%. The physics world went wild and hundreds of articles were published about it! It seems to have been resolved now, though.
Even if everything is in principle calculable, it doesn’t mean you can do useful calculations of complex systems a useful distance into the future. The three body problem intervenes. And there are rather more than three bodies if you’re trying to predict behavior of a brain-sized neural network, let alone intervening on a complex physical world. The computer you’d need wouldn’t just be the size of the universe, but all of the many worlds branches.
Simulation of the time evolution of models from their dynamical equations is only one way of proving properties about them. For example, a harmonic oscillator https://en.wikipedia.org/wiki/Harmonic_oscillator has dynamical equations m d^2x/dt^2= -kx. You can simulate that but you can also prove that the kinetic plus potential energy is conserved and get limits on its behavior arbitrarily far into the future.
Sure but seems highly unlikely there are any such neat simplifications for complex cognitive systems built from neural networks.
Other than “sapient beings do things that further their goals in their best estimation”, which is a rough predictor, and what we’re already trying to focus on. But the devil is in the details, and the important question is about how the goal is represented and understood.
Oh yeah, by their very nature it’s likely to be hard to predict intelligent systems behavior in detail. We can put constraints on them, though, and prove that they operate within those constraints.
Even simple systems like random SAT problems https://en.wikipedia.org/wiki/SAT_solver can have a very rich statistical structure. And the behavior of the solvers can be quite unpredictable.
In some sense, this is the source of unpredictability of cryptographic hash functions. Odet Goldreich proposed an unbelivable simple boolean function which is believed to be one-way: https://link.springer.com/chapter/10.1007/978-3-642-22670-0_10
On the other hand, I think it is often possible to distill behavior for a particlular task from a rich intelligence into simple code with provable properties.
(Mod note: Edited in the image)