One of the major problems I see regarding the power of a singleton superintelligence is that unknown unknowns are not sign-posted.
If people like Benoît B. Mandelbrot would have never decided to research Fractals then many modern movies wouldn’t be possible, as they rely on fractal landscape algorithms. Yet, at the time Benoît B. Mandelbrot conducted his research it was not foreseeable that his work would have any real-world applications.
Important discoveries are made because many routes with low or no expected utility are explored at the same time. And to do so efficiently it takes random mutation, a whole society of minds, a lot of feedback and empirical experimentation.
“Treating rare diseases in cute kittens” might or might not provide genuine insights and open up new avenues for further research. As long as you don’t try it you won’t know.
The idea that a rigid consequentialist with simple values can think up insights and conceptual revolutions simply because it is instrumentally useful to do so seems implausible to me.
Complex values are the cornerstone of diversity, which in turn enables creativity and drives the exploration of various conflicting routes. A singleton with a stable utility-function lacks the feedback provided by a society of minds and its cultural evolution.
If people like Benoît B. Mandelbrot would have never decided to research Fractals then many modern movies wouldn’t be possible, as they rely on fractal landscape algorithms. Yet, at the time Benoît B. Mandelbrot conducted his research it was not foreseeable that his work would have any real-world applications.
In many ways, this is a book about hindsight. Pythagoras could not have imagined the uses to which his equation would be put (if, indeed, he ever came up with the equation himself in the first place). The same applies to almost all of the equations in this book. They were studied/discovered/developed by mathematicians and mathematical physicists who were investigating subjects that fascinated them deeply, not because they imagined that two hundred years later the work would lead to electric light bulbs or GPS or the internet, but rather because they were genuinely curious.
Here is my list of “really stupid, frivolous academic pursuits” that have lead to major scientific breakthroughs.
Studying monkey social behaviors and eating habits lead to insights into HIV (Radiolab: Patient Zero)
Research into how algae move toward light paved the way for optogenetics: using light to control brain cells (Nature 2010 Method of the Year).
Black hole research gave us WiFi (ICRAR award)
Optometry informs architecture and saved lives on 9/11 (APA Monitor)
Certain groups HATE SETI, but SETI’s development of cloud-computing service SETI@HOME paved the way for citizen science and recent breakthroughs in protein folding (Popular Science)
Astronomers provide insights into medical imaging (TEDxBoston: Michell Borkin)
Basic physics experiments and the Fibonacci sequence help us understand plant growth and neuron development
I agree that a sufficiently simple-valued optimizer pretty much destroys most of what I value in any environment it controls.
You are implicitly equating “singleton” and “simple values” here in a way that doesn’t seem at all justified to me.
I agree that for all X, as long as you don’t try X, you don’t know what X might lead to. I would also add that for all X, even if you do try X, you don’t know what X might lead to. (For example, suppose I spend ten years working on a research project. It doesn’t follow that I know what spending ten years on that research project leads to; perhaps if I’d gone about it slightly differently, I’d have gotten different results.)
I agree that when we can’t think of a high-expected-utility route, we try low-expected-utility routes, because that’s what there is. And if enough of us do that, we often discover unexpected utility on those routes. That said, if there’s two routes I can take, and path A has a high chance of getting me what I want, and path B has a low chance of getting me what I want, I take path A.. So does basically every higher mammal I’ve ever met.
I agree that unlike mammals, self-replicating DNA with sources of random mutation are as likely to explore path A as path B. I don’t think it’s a coincidence that mammals as a class achieve their goals faster than self-replicating DNA with sources of random mutation.
You are implicitly equating “singleton” and “simple values” here in a way that doesn’t seem at all justified to me.
No, I don’t. What I am saying is that you need to have various different agents with different utility-functions around to get the necessary diversity that can give rise to enough selection pressure. I am further saying that a “singleton” won’t be able to predict the actions of new and improved versions of itself by just running sandboxed simulations. Not just because of logical uncertainty but also because it is computationally intractable to predict the real-world payoff of changes to its decision procedures.
I am also saying that you need complex values to give rise to the necessary drives to function in a complex world. You can’t just tell an AI to protect itself. What would that even mean? What changes are illegitimate? What constitutes “self”? That are all unsolved problems that are just assumed to be solvable when talking about risks from AI.
...when we can’t think of a high-expected-utility route, we try low-expected-utility routes, because that’s what there is. And if enough of us do that, we often discover unexpected utility on those routes. That said, if there’s two routes I can take, and path A has a high chance of getting me what I want, and path B has a low chance of getting me what I want, I take path A..
What I am talking about is concurrence. What I claim won’t work is the kind of arguments put forth by people like Steven Landsburg that you should contribute to just one charity that you deem most important. The real world is not that simple. Much progress is made due to unpredictable synergy. “Treating rare diseases in cute kittens” might lead to insights on solving cancer in humans.
If you are an AI with simple values you will simply lack the creativity, due to a lack of drives, to pursue the huge spectrum of research that a society of humans does pursue. Which will allow an AI to solve some well-defined narrow problems, but it will be unable to make use of the broad range of synergetic effects of cultural evolution. Cultural evolution is a result of the interaction of a wide range of utility-functions.
I agree that unlike mammals, self-replicating DNA with sources of random mutation are as likely to explore path A as path B. I don’t think it’s a coincidence that mammals as a class achieve their goals faster than self-replicating DNA with sources of random mutation.
The difference is that mammals have goals, complex values, which allows them to make use of evolutionary discoveries and adapt them for their purposes.
One of the major problems I see regarding the power of a singleton superintelligence is that unknown unknowns are not sign-posted.
If people like Benoît B. Mandelbrot would have never decided to research Fractals then many modern movies wouldn’t be possible, as they rely on fractal landscape algorithms. Yet, at the time Benoît B. Mandelbrot conducted his research it was not foreseeable that his work would have any real-world applications.
Important discoveries are made because many routes with low or no expected utility are explored at the same time. And to do so efficiently it takes random mutation, a whole society of minds, a lot of feedback and empirical experimentation.
“Treating rare diseases in cute kittens” might or might not provide genuine insights and open up new avenues for further research. As long as you don’t try it you won’t know.
The idea that a rigid consequentialist with simple values can think up insights and conceptual revolutions simply because it is instrumentally useful to do so seems implausible to me.
Complex values are the cornerstone of diversity, which in turn enables creativity and drives the exploration of various conflicting routes. A singleton with a stable utility-function lacks the feedback provided by a society of minds and its cultural evolution.
Addendum (via 17 Equations that changed the world)
(emphasis mine)
Addendum (via Basic science is about creating opportunities)
Studying monkey social behaviors and eating habits lead to insights into HIV (Radiolab: Patient Zero)
Research into how algae move toward light paved the way for optogenetics: using light to control brain cells (Nature 2010 Method of the Year).
Black hole research gave us WiFi (ICRAR award)
Optometry informs architecture and saved lives on 9/11 (APA Monitor)
Certain groups HATE SETI, but SETI’s development of cloud-computing service SETI@HOME paved the way for citizen science and recent breakthroughs in protein folding (Popular Science)
Astronomers provide insights into medical imaging (TEDxBoston: Michell Borkin)
Basic physics experiments and the Fibonacci sequence help us understand plant growth and neuron development
(References)
I agree that a sufficiently simple-valued optimizer pretty much destroys most of what I value in any environment it controls.
You are implicitly equating “singleton” and “simple values” here in a way that doesn’t seem at all justified to me.
I agree that for all X, as long as you don’t try X, you don’t know what X might lead to. I would also add that for all X, even if you do try X, you don’t know what X might lead to. (For example, suppose I spend ten years working on a research project. It doesn’t follow that I know what spending ten years on that research project leads to; perhaps if I’d gone about it slightly differently, I’d have gotten different results.)
I agree that when we can’t think of a high-expected-utility route, we try low-expected-utility routes, because that’s what there is. And if enough of us do that, we often discover unexpected utility on those routes. That said, if there’s two routes I can take, and path A has a high chance of getting me what I want, and path B has a low chance of getting me what I want, I take path A.. So does basically every higher mammal I’ve ever met.
I agree that unlike mammals, self-replicating DNA with sources of random mutation are as likely to explore path A as path B. I don’t think it’s a coincidence that mammals as a class achieve their goals faster than self-replicating DNA with sources of random mutation.
No, I don’t. What I am saying is that you need to have various different agents with different utility-functions around to get the necessary diversity that can give rise to enough selection pressure. I am further saying that a “singleton” won’t be able to predict the actions of new and improved versions of itself by just running sandboxed simulations. Not just because of logical uncertainty but also because it is computationally intractable to predict the real-world payoff of changes to its decision procedures.
I am also saying that you need complex values to give rise to the necessary drives to function in a complex world. You can’t just tell an AI to protect itself. What would that even mean? What changes are illegitimate? What constitutes “self”? That are all unsolved problems that are just assumed to be solvable when talking about risks from AI.
What I am talking about is concurrence. What I claim won’t work is the kind of arguments put forth by people like Steven Landsburg that you should contribute to just one charity that you deem most important. The real world is not that simple. Much progress is made due to unpredictable synergy. “Treating rare diseases in cute kittens” might lead to insights on solving cancer in humans.
If you are an AI with simple values you will simply lack the creativity, due to a lack of drives, to pursue the huge spectrum of research that a society of humans does pursue. Which will allow an AI to solve some well-defined narrow problems, but it will be unable to make use of the broad range of synergetic effects of cultural evolution. Cultural evolution is a result of the interaction of a wide range of utility-functions.
The difference is that mammals have goals, complex values, which allows them to make use of evolutionary discoveries and adapt them for their purposes.