I think there probably is a good reference class for predictions surrounding the singularity. When you posted on “what is wrong with our thoughts? you identified it: the class of instances of the human mind attempting to think and act outside of its epistemologically nurturing environment of clear feedback from everyday activities.
See, e.g. how smart humans like Stephen Hawking, Ray Kurzweil, Kevin Warwick, Kevin Kelly, Eric Horowitz, etc have all managed to say patently absurd things about the issue, and hold mutually contradictory positions, with massive overconfidence in some cases. I do not exclude myself from the group of people who have said absurd things about the Singularity, and I think we shouldn’t exclude Eliezer either. At least Eliezer has put in massive amounts of work for what may well be the greater good of humanity, which is morally commendable.
To escape from this reference class, and therefore from the default prediction of insanity, I think that bringing in better feedback and a large diverse community of researchers might work. Of course, more feedback and more researchers = more risk according to our understanding of AI motivations. But ultimately, that’s an unavoidable trade-off; the lone madman versus the global tragedy of the commons.
Large communities don’t constitute help or progress on the “beyond the realm of feedback” problem. In the absence of feedback, how is a community supposed to know when one of its members has made progress? Even with feedback we have cases like psychotherapy and dietary science where experimental results are simply ignored. Look at the case of physics and many-worlds. What has “diversity” done for the Singularity so far? Kurzweil has gotten more people talking about “the Singularity”—and lo, the average wit of the majority hath fallen. If anything, trying to throw a large community at the problem just guarantees that you get the average result of failure, rather than being able to notice one of the rare individuals or minority communities that can make progress using lower amounts of evidence.
I may even go so far as to call “applause light” or “unrelated charge of positive affect” on the invocation of a “diverse community” here, because of the degree to which the solution fails to address the problem.
In the absence of feedback, how is a community supposed to know when one of its members has made progress?
Good question. It seems that academic philosophy does, to an extent, achieve this. The mechanism seems to be that it is easier to check an argument for correctness than to generate it. And it is easier to check whether a claimed flaw in an argument really is a flaw, and so on.
In this case, a mechanism where everyone in the community tries to think of arguments, and tries to think of flaws in others’ arguments, and tries to think of flaws in the criticisms of arguments, etc, means that as the community size --> infinity, the field converges on the truth.
Good question. It seems that academic philosophy does, to an extent, achieve this.
With some of my engagements with academic philosophers in mind I have at times been tempted to lament that that ‘extent’ wasn’t rather a lot greater. Of course, that may be ‘the glass is half empty’ thinking. I intuit that there is potential for a larger body of contributers to have even more of a correcting influence of the kind that you mention than what we see in practice!
Philosophy has made some pretty significant progress in many areas. However, sometimes disciplines of that form can get “stuck” in an inescapable pit of nonsense, e.g. postmodernism or theology. In a sense, the philosophy community is trying to re-do what the theologians have failed at: answering questions such as “how should I live”, etc.
Many-worlds has made steady progress since it was invented. Especially early on, trying to bring in diversity would get you some many-worlds proponents rather than none, and their views would tend to spread.
Think of how much more progress could have been made if the early many-worlds proponents had gotten together and formed a private colloquium of the sane, providing only that they had access to the same amount of per capita grant funding (this latter point being not about a need for diversity but a need to pander to gatekeepers).
However I wouldn’t go the lots of people route either. At least not until decent research norms had been created.
The research methodology that has been mouldering away in my brain for the past few years is the following:
We can agree that computational systems might be dangerous (in the FOOM sense).
So let us start from the basics and prove that bits of computer space aren’t dangerous either by experiments we have already done (or have been done by nature) or by formal proof.
Humanity has played around with basic computers and networked computers in a variety of configurations, if our theories say that they are dangerous then our theories are probably wrong.
Nature has and is also in the process of creating many computational systems. The Gene networks I mentioned earlier and if you want to look at the air around you as a giant quantum billiard ball computer of sorts, then giant ephemeral networks of “if molecule A collides with molecule B then molecule A will collide with molecule C” type calculations are being performed all around you without danger. *
The proof section is more controversial. There are certain mathematical properties I would expect powerful systems to have. The ability to accept recursive languages and also modify internal state (especially state that controls how they accept languages) based on them seems crucial to me. If we could build up a list of properties like this we can prove that certain systems don’t have them and aren’t going to be dangerous.
You can also correlate the dangerousness of parts of computational space with other parts of computational space.
One way of looking at self-modifying systems is that it that they are equivalent to non-self-modifying systems with infinite program memory and a bit of optimisation. As if you can write a program that changes function X to Y when it sees input Z, you can write a program that chooses to perform function X rather than Y if input Z has been seen using a simple if condition. Of course as you add more and more branches you increase the memory needed to store all the different programs. However I haven’t seen anyone put bounds on how large the memory would have to be for the system to be considered dangerous. It might well be larger than the universe, but that would be good to know.
In this way we can whittle down where is dangerous until we either prove that
a) There is FOOMy part of computer space
b) There is a non-FOOMy part of AI space, so we can build that and use that to try and figure out how to avoid malthusian scenarios and whittle down the space more
c) We have a space which we can’t prove much about. We should have a respectable science to say, “here might be dragons,” by this point.
d) We cover all of the computer space and none of it is FOOMy.
My personal bet is on b. But I am just a monkey...
Edit: Re-lurk. I have to be doing other things to do with boring surviving so don’t expect much out of me for a good few months.
*An interesting question is do any of these network implement the calculations for pain/pleasure or other qualia.
I think you may be overestimating how much work formal proof can do here. For example, could formal proof have proved that early homonids would cause the human explosion?
Data about the world is very important in my view of intelligence.
Hominid brains were collecting lots of information about the world, then losing it all when they were dying, because they couldn’t pass it all on. They could only pass on what they could demonstrate directly. (Lots of other species were doing so as well, so this argument applies to them as well.)
The species that managed to keep a hold of this lost information and spread it far and wide, you could probably prove would have a different learning pattern to the “start from scratch-learn/mimic-die” model of most animals, and potentially explode as “things with brains” had before.
Could you have proven it would be homonids? Possibly, you would need to know more about how the systems could realistically spread information between them including protection from lying and manipulation. And whether homonids had the properties that made them more likely to explode.
I think there probably is a good reference class for predictions surrounding the singularity. When you posted on “what is wrong with our thoughts? you identified it: the class of instances of the human mind attempting to think and act outside of its epistemologically nurturing environment of clear feedback from everyday activities.
See, e.g. how smart humans like Stephen Hawking, Ray Kurzweil, Kevin Warwick, Kevin Kelly, Eric Horowitz, etc have all managed to say patently absurd things about the issue, and hold mutually contradictory positions, with massive overconfidence in some cases. I do not exclude myself from the group of people who have said absurd things about the Singularity, and I think we shouldn’t exclude Eliezer either. At least Eliezer has put in massive amounts of work for what may well be the greater good of humanity, which is morally commendable.
To escape from this reference class, and therefore from the default prediction of insanity, I think that bringing in better feedback and a large diverse community of researchers might work. Of course, more feedback and more researchers = more risk according to our understanding of AI motivations. But ultimately, that’s an unavoidable trade-off; the lone madman versus the global tragedy of the commons.
Large communities don’t constitute help or progress on the “beyond the realm of feedback” problem. In the absence of feedback, how is a community supposed to know when one of its members has made progress? Even with feedback we have cases like psychotherapy and dietary science where experimental results are simply ignored. Look at the case of physics and many-worlds. What has “diversity” done for the Singularity so far? Kurzweil has gotten more people talking about “the Singularity”—and lo, the average wit of the majority hath fallen. If anything, trying to throw a large community at the problem just guarantees that you get the average result of failure, rather than being able to notice one of the rare individuals or minority communities that can make progress using lower amounts of evidence.
I may even go so far as to call “applause light” or “unrelated charge of positive affect” on the invocation of a “diverse community” here, because of the degree to which the solution fails to address the problem.
Good question. It seems that academic philosophy does, to an extent, achieve this. The mechanism seems to be that it is easier to check an argument for correctness than to generate it. And it is easier to check whether a claimed flaw in an argument really is a flaw, and so on.
In this case, a mechanism where everyone in the community tries to think of arguments, and tries to think of flaws in others’ arguments, and tries to think of flaws in the criticisms of arguments, etc, means that as the community size --> infinity, the field converges on the truth.
With some of my engagements with academic philosophers in mind I have at times been tempted to lament that that ‘extent’ wasn’t rather a lot greater. Of course, that may be ‘the glass is half empty’ thinking. I intuit that there is potential for a larger body of contributers to have even more of a correcting influence of the kind that you mention than what we see in practice!
Philosophy has made some pretty significant progress in many areas. However, sometimes disciplines of that form can get “stuck” in an inescapable pit of nonsense, e.g. postmodernism or theology. In a sense, the philosophy community is trying to re-do what the theologians have failed at: answering questions such as “how should I live”, etc.
Many-worlds has made steady progress since it was invented. Especially early on, trying to bring in diversity would get you some many-worlds proponents rather than none, and their views would tend to spread.
Think of how much more progress could have been made if the early many-worlds proponents had gotten together and formed a private colloquium of the sane, providing only that they had access to the same amount of per capita grant funding (this latter point being not about a need for diversity but a need to pander to gatekeepers).
It isn’t clear to me that the MWI-only group would have achieved anything extra—do you think that they would have done?
Logged in to vote this up...
However I wouldn’t go the lots of people route either. At least not until decent research norms had been created.
The research methodology that has been mouldering away in my brain for the past few years is the following:
We can agree that computational systems might be dangerous (in the FOOM sense).
So let us start from the basics and prove that bits of computer space aren’t dangerous either by experiments we have already done (or have been done by nature) or by formal proof.
Humanity has played around with basic computers and networked computers in a variety of configurations, if our theories say that they are dangerous then our theories are probably wrong.
Nature has and is also in the process of creating many computational systems. The Gene networks I mentioned earlier and if you want to look at the air around you as a giant quantum billiard ball computer of sorts, then giant ephemeral networks of “if molecule A collides with molecule B then molecule A will collide with molecule C” type calculations are being performed all around you without danger. *
The proof section is more controversial. There are certain mathematical properties I would expect powerful systems to have. The ability to accept recursive languages and also modify internal state (especially state that controls how they accept languages) based on them seems crucial to me. If we could build up a list of properties like this we can prove that certain systems don’t have them and aren’t going to be dangerous.
You can also correlate the dangerousness of parts of computational space with other parts of computational space.
One way of looking at self-modifying systems is that it that they are equivalent to non-self-modifying systems with infinite program memory and a bit of optimisation. As if you can write a program that changes function X to Y when it sees input Z, you can write a program that chooses to perform function X rather than Y if input Z has been seen using a simple if condition. Of course as you add more and more branches you increase the memory needed to store all the different programs. However I haven’t seen anyone put bounds on how large the memory would have to be for the system to be considered dangerous. It might well be larger than the universe, but that would be good to know.
In this way we can whittle down where is dangerous until we either prove that
a) There is FOOMy part of computer space b) There is a non-FOOMy part of AI space, so we can build that and use that to try and figure out how to avoid malthusian scenarios and whittle down the space more c) We have a space which we can’t prove much about. We should have a respectable science to say, “here might be dragons,” by this point. d) We cover all of the computer space and none of it is FOOMy.
My personal bet is on b. But I am just a monkey...
Edit: Re-lurk. I have to be doing other things to do with boring surviving so don’t expect much out of me for a good few months.
*An interesting question is do any of these network implement the calculations for pain/pleasure or other qualia.
Thanks!
I think you may be overestimating how much work formal proof can do here. For example, could formal proof have proved that early homonids would cause the human explosion?
Data about the world is very important in my view of intelligence.
Hominid brains were collecting lots of information about the world, then losing it all when they were dying, because they couldn’t pass it all on. They could only pass on what they could demonstrate directly. (Lots of other species were doing so as well, so this argument applies to them as well.)
The species that managed to keep a hold of this lost information and spread it far and wide, you could probably prove would have a different learning pattern to the “start from scratch-learn/mimic-die” model of most animals, and potentially explode as “things with brains” had before.
Could you have proven it would be homonids? Possibly, you would need to know more about how the systems could realistically spread information between them including protection from lying and manipulation. And whether homonids had the properties that made them more likely to explode.