However I wouldn’t go the lots of people route either. At least not until decent research norms had been created.
The research methodology that has been mouldering away in my brain for the past few years is the following:
We can agree that computational systems might be dangerous (in the FOOM sense).
So let us start from the basics and prove that bits of computer space aren’t dangerous either by experiments we have already done (or have been done by nature) or by formal proof.
Humanity has played around with basic computers and networked computers in a variety of configurations, if our theories say that they are dangerous then our theories are probably wrong.
Nature has and is also in the process of creating many computational systems. The Gene networks I mentioned earlier and if you want to look at the air around you as a giant quantum billiard ball computer of sorts, then giant ephemeral networks of “if molecule A collides with molecule B then molecule A will collide with molecule C” type calculations are being performed all around you without danger. *
The proof section is more controversial. There are certain mathematical properties I would expect powerful systems to have. The ability to accept recursive languages and also modify internal state (especially state that controls how they accept languages) based on them seems crucial to me. If we could build up a list of properties like this we can prove that certain systems don’t have them and aren’t going to be dangerous.
You can also correlate the dangerousness of parts of computational space with other parts of computational space.
One way of looking at self-modifying systems is that it that they are equivalent to non-self-modifying systems with infinite program memory and a bit of optimisation. As if you can write a program that changes function X to Y when it sees input Z, you can write a program that chooses to perform function X rather than Y if input Z has been seen using a simple if condition. Of course as you add more and more branches you increase the memory needed to store all the different programs. However I haven’t seen anyone put bounds on how large the memory would have to be for the system to be considered dangerous. It might well be larger than the universe, but that would be good to know.
In this way we can whittle down where is dangerous until we either prove that
a) There is FOOMy part of computer space
b) There is a non-FOOMy part of AI space, so we can build that and use that to try and figure out how to avoid malthusian scenarios and whittle down the space more
c) We have a space which we can’t prove much about. We should have a respectable science to say, “here might be dragons,” by this point.
d) We cover all of the computer space and none of it is FOOMy.
My personal bet is on b. But I am just a monkey...
Edit: Re-lurk. I have to be doing other things to do with boring surviving so don’t expect much out of me for a good few months.
*An interesting question is do any of these network implement the calculations for pain/pleasure or other qualia.
I think you may be overestimating how much work formal proof can do here. For example, could formal proof have proved that early homonids would cause the human explosion?
Data about the world is very important in my view of intelligence.
Hominid brains were collecting lots of information about the world, then losing it all when they were dying, because they couldn’t pass it all on. They could only pass on what they could demonstrate directly. (Lots of other species were doing so as well, so this argument applies to them as well.)
The species that managed to keep a hold of this lost information and spread it far and wide, you could probably prove would have a different learning pattern to the “start from scratch-learn/mimic-die” model of most animals, and potentially explode as “things with brains” had before.
Could you have proven it would be homonids? Possibly, you would need to know more about how the systems could realistically spread information between them including protection from lying and manipulation. And whether homonids had the properties that made them more likely to explode.
Logged in to vote this up...
However I wouldn’t go the lots of people route either. At least not until decent research norms had been created.
The research methodology that has been mouldering away in my brain for the past few years is the following:
We can agree that computational systems might be dangerous (in the FOOM sense).
So let us start from the basics and prove that bits of computer space aren’t dangerous either by experiments we have already done (or have been done by nature) or by formal proof.
Humanity has played around with basic computers and networked computers in a variety of configurations, if our theories say that they are dangerous then our theories are probably wrong.
Nature has and is also in the process of creating many computational systems. The Gene networks I mentioned earlier and if you want to look at the air around you as a giant quantum billiard ball computer of sorts, then giant ephemeral networks of “if molecule A collides with molecule B then molecule A will collide with molecule C” type calculations are being performed all around you without danger. *
The proof section is more controversial. There are certain mathematical properties I would expect powerful systems to have. The ability to accept recursive languages and also modify internal state (especially state that controls how they accept languages) based on them seems crucial to me. If we could build up a list of properties like this we can prove that certain systems don’t have them and aren’t going to be dangerous.
You can also correlate the dangerousness of parts of computational space with other parts of computational space.
One way of looking at self-modifying systems is that it that they are equivalent to non-self-modifying systems with infinite program memory and a bit of optimisation. As if you can write a program that changes function X to Y when it sees input Z, you can write a program that chooses to perform function X rather than Y if input Z has been seen using a simple if condition. Of course as you add more and more branches you increase the memory needed to store all the different programs. However I haven’t seen anyone put bounds on how large the memory would have to be for the system to be considered dangerous. It might well be larger than the universe, but that would be good to know.
In this way we can whittle down where is dangerous until we either prove that
a) There is FOOMy part of computer space b) There is a non-FOOMy part of AI space, so we can build that and use that to try and figure out how to avoid malthusian scenarios and whittle down the space more c) We have a space which we can’t prove much about. We should have a respectable science to say, “here might be dragons,” by this point. d) We cover all of the computer space and none of it is FOOMy.
My personal bet is on b. But I am just a monkey...
Edit: Re-lurk. I have to be doing other things to do with boring surviving so don’t expect much out of me for a good few months.
*An interesting question is do any of these network implement the calculations for pain/pleasure or other qualia.
Thanks!
I think you may be overestimating how much work formal proof can do here. For example, could formal proof have proved that early homonids would cause the human explosion?
Data about the world is very important in my view of intelligence.
Hominid brains were collecting lots of information about the world, then losing it all when they were dying, because they couldn’t pass it all on. They could only pass on what they could demonstrate directly. (Lots of other species were doing so as well, so this argument applies to them as well.)
The species that managed to keep a hold of this lost information and spread it far and wide, you could probably prove would have a different learning pattern to the “start from scratch-learn/mimic-die” model of most animals, and potentially explode as “things with brains” had before.
Could you have proven it would be homonids? Possibly, you would need to know more about how the systems could realistically spread information between them including protection from lying and manipulation. And whether homonids had the properties that made them more likely to explode.