In reading the Singularity Institute’s research goals, and the ruminations of Yudkowski, Wei Dai, Nesov et al. in postings here, the approach to developing friendly AI which stands out the most, and from my perspective seems to just always have been the case, seems to be exclusively logic based in the vein of John McCarthy.
I am wondering how the decision was made to focus research for SIAI on the pure logic side, rather than, for example building a synthetic conscious which uses the brain as a model?
To be sure, nearly all AI approaches overlap at some point. However it seems a bit like reinventing the wheel from an almost purely theoretical perspective, when you have a very good model to work from.
Let’s say you’ve got a prototype you want to improve. How do you tell if a proposed change would make it smarter, break it, introduce a subtle cognitive bias, or make the AI want to kill you?
In order to set on limits on the kinds of things an AI will do, you need to understand how it works. You can’t be experimenting on a structure you partially understand, AND be certain that the experiments won’t be fatal.
This is easier when you’ve got a clearly defined structure to the AI, and know how the parts interact, and why.
In order to set on limits on the kinds of things an AI will do, you need to understand how it works.
How is that impossible with a replicated brain architecture? We can’t make one if we don’t know how it works.
This is easier when you’ve got a clearly defined structure to the AI, and know how the parts interact, and why.
Of course. However, how you plan to structure AI what I am asking about. There are many theories about how to structure the AI—so why did the SIAI choose to only focus on a theoretical mathematical logic based approach rather than taking the most advanced, if still flawed, logic device known to man and replicating and improving that?
In order to set on limits on the kinds of things an AI will do, you need to understand how it works.
How is that impossible with a replicated brain architecture? We can’t make one if we don’t know how it works.
If you have the right tools, you can make a brain without understanding it. Reproductive system can make brains. Whole brain emulation doesn’t require understanding of brain architecture, only the dynamics of its lowest-level components.
You “know” how pi works, and how to set up a program that computes it, but you don’t know what its quandrillionth digit is.
Whole brain emulation doesn’t require understanding of brain architecture, only the dynamics of its lowest-level components.
I fear the same philosophical reasoning may be applied to model neural architecture as is currently being used for econometric forecasting. Even the most complex economic models cannot account for significant exogenous variables.
For the record I think we can get to WBE, however I think a premature launch would be terrible. Based on the lack of research into developmental AI (much work notably done by a friend—Dr. Frank Guerin at Aberdeen college) I think there is a long way to go.
Granting that a brain model or WBE, would be as accurate as the biological version, why then would that not be the most efficient method? The problems with testing and implementation are the same as any other AI, if not easier because of familiarity, however it is grounded on specific biological benchmarks which at that point would be immediately identifiable.
I could go on with my particular thoughts as to why biological simulation is in my estimation a better approach, however I am more interested in why the organization (people who have been thinking longer and with more effort than myself) decided otherwise. It would seem that their collective reasoning would give a sufficiently clear and precise answer such that there would be no ambiguity.
Granting that a brain model or WBE, would be as accurate as the biological version, why then would that not be the most efficient method?
You have to instill the right preference, and just having a working improved brain doesn’t give this capability. You are trying to make an overwhelmingly powerful ally; just making something overwhelmingly powerful is a suicide. As CannibalSmith said, brains are not particularly Friendly. Read the paper.
You have to instill the right preference, and just having a working improved brain doesn’t give this capability.
Of course—we have to BUILD IT RIGHT. I couldn’t agree more. The cognitive model does not suggest a mere carbon copy of any particular brain at random, as you know it is not so limited in focus. The fantastic part about the method is at the point in which it is possible to do correctly (not simply an apparent approximation), the tools will likely be available (in the process of it being structured) to correct a large portion of what we identify as fatal cognitive errors. Any errors that are missed it stands to reason would be also missed given the same amount of time with any other developmental structure.
I am familiar with the global risk paper you linked, AI: A modern approach which addresses the issue of cognitive modeling as well as Drescher’s Good and Real and the problems associated with an FAI.
The same existential risks and potential for human disasters are inherent in all AI systems—regardless of the structure, by virtue of it’s “power.” I think one of the draws to this type of development is the fantastic responsibility which comes with it’s development, recognizing and accounting for the catastrophic results that are possible.
That said, I have yet to read a decision theoretic explication as to which structure is an optimized method of development, weighing all known limiting factors. I think AI: A modern approach comes closest to doing this but falls short in that it specifically narrows it’s focus without a thorough comparison of methods. So again, I ask, by what construct has it been determined that a logical symbolic programming approach is optimized?
In other words they are doing it where the light’s better, rather than where they dropped the keys. Given the track record of correctness proofs in comp sci, I don’t think provably Friendly AI is even possible, hopefully I’m wrong there, but all they are doing is further crippling their likelihood of achieving AI before some military or business does.
People control how companies operate—even though they don’t understand all the components involved (in particular the brains).
Any idea that you have to understand all the components of a system in order to exert a high level of control over it thus appears to have dubious foundations.
There are alternative approaches to producing predictable systems which basically involve lots of unit tests.
Testing is in vogue in the software engineering world. Few systems are simple enough to prove much about their behaviour. So: to make sure they behave as they are intended, they are intensively tested. It seems likely that machine intelligence will be no different.
This would be true of any AI. Thus the AI box problem.
It is unclear however, how a formal logic approach overcomes this problem and a replication approach does not. They both will need testing, but as you said the methodology should be reconsidered. The easiest way to change testing methodology for logic would be to improve on current logic methodology which has yielded arguably fantastic results—all done by faulty human brains.
Normally testing is done in an offline “testing” mode—using a test harness or sandbox arrangement. Tests themselves are consequently harmless.
Of course it is possible for the world to present eventualities that are not modelled by the test suite—but that’s usually no big deal.
I don’t think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product.
I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM
will be widely ignored. This isn’t exactly the first time that people have claimed that the world is going to end.
Forget about whether your sandbox is a realistic enough test. There are even questions about how much safety you’re getting from a sandbox. So, we follow your advice, and put the AI in a box in order to test it. And then it escapes anyway, during the test.
The idea that society is smart enough to build machine intelligence, but not smart enough to build a box to test it in does not seem credible to me:
Humans build boxes to put other humans in—and have a high success rate of keeping them inside when they put their minds to it. The few rogue agents that do escape are typically hunted down and imprisoned again. Basically the builders of the box are much stronger and more powerful than what it will contain. Machine intelligence testing seems unlikely to be significantly different from that situation.
The cited “box” scenario discusses the case of weak gatekeepers and powerful escapees. That scenario isn’t very relevant in this case—since we will have smart machines on both sides when restraining intelligent machines in order to test them.
Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.
DOOM is actually big business. Check out all the disaster movies out there. DOOM sells. What could be more important than… THE END OF THE WORLD? What greater cause could there be than… SAVING THE WORLD? So, people buy the DOOM merchandise, contribute the DOOM dollars, and warn their friends about the impending DOOM—thus perpetuating the DOOM virus. That is part of why there have been so many DOOM prophets—DOOM pays.
I would disagree. The overwhelming majority of the average human’s life is spent peacefully. It is actually fairly remarkable how rarely we have significant conflict, especially considering the relatively overcrowded places that humans live. Not to mention that it is only a small proportion of the human population that engages other humans destructively (not by proxy).
The overwhelming majority of the average human’s life is also spent in conditions where they are on relatively even grounds with everyone else. But once you start looking at what happens when people end up in situations where they are clearly more powerful than others? And can treat those others the way they like and without fear of retribution? Ugly.
I disagree. While there are some spectacular examples of what you describe, and they are indeed ugly, by and large there is wide distribution of hierarchical disparity even in daily life which is more often than not mutually beneficent.
As an emperor I optimize my empire by ensuring that my subjects are philosophically and physically satisfied do I not? I think there is plenty of evidence to support this philosophy as the most sustainable (and positive) of hierarchical models; after all some of the most successful businesses are laterally organized.
A certain philosophy being the most sustainable and positive isn’t automatically the same as being the one people tend to adopt. Plus the answer to your question depends on what you’re trying to optimize.
Also, it sounds like you’re still talking about a situation where people don’t actually have ultimate power. If we’re discussing a potential hard takeoff scenario, then considerations such as “which models have been the most successful for businesses before” don’t really apply. Any entity genuinely undergoing a hard takeoff is one that isn’t afterwards bound by what’s successful for humans, any more than we are bound by the practices that work the best for ants.
A certain philosophy being the most sustainable and positive isn’t automatically the same as being the one people tend to adopt
I think there is more than ample evidence to suggest that those are significantly less likely to be adopted—however wouldn’t a group of people who know that and can correct for it be the best test case of implementing an optimized strategy?
Also, it sounds like you’re still talking about a situation where people don’t actually have ultimate power.
I hold the view that it is unnecessary to hold ultimate power over FAI. I certainly wouldn’t bind it to what has worked for humans thus far. Don’t fear the AI, find a way to assimilate.
In reading the Singularity Institute’s research goals, and the ruminations of Yudkowski, Wei Dai, Nesov et al. in postings here, the approach to developing friendly AI which stands out the most, and from my perspective seems to just always have been the case, seems to be exclusively logic based in the vein of John McCarthy.
I am wondering how the decision was made to focus research for SIAI on the pure logic side, rather than, for example building a synthetic conscious which uses the brain as a model?
To be sure, nearly all AI approaches overlap at some point. However it seems a bit like reinventing the wheel from an almost purely theoretical perspective, when you have a very good model to work from.
Such a design would be harder to reason about.
Let’s say you’ve got a prototype you want to improve. How do you tell if a proposed change would make it smarter, break it, introduce a subtle cognitive bias, or make the AI want to kill you?
In order to set on limits on the kinds of things an AI will do, you need to understand how it works. You can’t be experimenting on a structure you partially understand, AND be certain that the experiments won’t be fatal.
This is easier when you’ve got a clearly defined structure to the AI, and know how the parts interact, and why.
How is that impossible with a replicated brain architecture? We can’t make one if we don’t know how it works.
Of course. However, how you plan to structure AI what I am asking about. There are many theories about how to structure the AI—so why did the SIAI choose to only focus on a theoretical mathematical logic based approach rather than taking the most advanced, if still flawed, logic device known to man and replicating and improving that?
If you have the right tools, you can make a brain without understanding it. Reproductive system can make brains. Whole brain emulation doesn’t require understanding of brain architecture, only the dynamics of its lowest-level components.
You “know” how pi works, and how to set up a program that computes it, but you don’t know what its quandrillionth digit is.
I fear the same philosophical reasoning may be applied to model neural architecture as is currently being used for econometric forecasting. Even the most complex economic models cannot account for significant exogenous variables.
For the record I think we can get to WBE, however I think a premature launch would be terrible. Based on the lack of research into developmental AI (much work notably done by a friend—Dr. Frank Guerin at Aberdeen college) I think there is a long way to go.
Granting that a brain model or WBE, would be as accurate as the biological version, why then would that not be the most efficient method? The problems with testing and implementation are the same as any other AI, if not easier because of familiarity, however it is grounded on specific biological benchmarks which at that point would be immediately identifiable.
I could go on with my particular thoughts as to why biological simulation is in my estimation a better approach, however I am more interested in why the organization (people who have been thinking longer and with more effort than myself) decided otherwise. It would seem that their collective reasoning would give a sufficiently clear and precise answer such that there would be no ambiguity.
You have to instill the right preference, and just having a working improved brain doesn’t give this capability. You are trying to make an overwhelmingly powerful ally; just making something overwhelmingly powerful is a suicide. As CannibalSmith said, brains are not particularly Friendly. Read the paper.
Of course—we have to BUILD IT RIGHT. I couldn’t agree more. The cognitive model does not suggest a mere carbon copy of any particular brain at random, as you know it is not so limited in focus. The fantastic part about the method is at the point in which it is possible to do correctly (not simply an apparent approximation), the tools will likely be available (in the process of it being structured) to correct a large portion of what we identify as fatal cognitive errors. Any errors that are missed it stands to reason would be also missed given the same amount of time with any other developmental structure.
I am familiar with the global risk paper you linked, AI: A modern approach which addresses the issue of cognitive modeling as well as Drescher’s Good and Real and the problems associated with an FAI.
The same existential risks and potential for human disasters are inherent in all AI systems—regardless of the structure, by virtue of it’s “power.” I think one of the draws to this type of development is the fantastic responsibility which comes with it’s development, recognizing and accounting for the catastrophic results that are possible.
That said, I have yet to read a decision theoretic explication as to which structure is an optimized method of development, weighing all known limiting factors. I think AI: A modern approach comes closest to doing this but falls short in that it specifically narrows it’s focus without a thorough comparison of methods. So again, I ask, by what construct has it been determined that a logical symbolic programming approach is optimized?
In other words they are doing it where the light’s better, rather than where they dropped the keys. Given the track record of correctness proofs in comp sci, I don’t think provably Friendly AI is even possible, hopefully I’m wrong there, but all they are doing is further crippling their likelihood of achieving AI before some military or business does.
People control how companies operate—even though they don’t understand all the components involved (in particular the brains).
Any idea that you have to understand all the components of a system in order to exert a high level of control over it thus appears to have dubious foundations.
There are alternative approaches to producing predictable systems which basically involve lots of unit tests.
Testing is in vogue in the software engineering world. Few systems are simple enough to prove much about their behaviour. So: to make sure they behave as they are intended, they are intensively tested. It seems likely that machine intelligence will be no different.
When a failed test destroys the world, applicability of the normally very useful testing methodology should be reconsidered.
This would be true of any AI. Thus the AI box problem.
It is unclear however, how a formal logic approach overcomes this problem and a replication approach does not. They both will need testing, but as you said the methodology should be reconsidered. The easiest way to change testing methodology for logic would be to improve on current logic methodology which has yielded arguably fantastic results—all done by faulty human brains.
Normally testing is done in an offline “testing” mode—using a test harness or sandbox arrangement. Tests themselves are consequently harmless.
Of course it is possible for the world to present eventualities that are not modelled by the test suite—but that’s usually no big deal.
I don’t think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product.
I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM will be widely ignored. This isn’t exactly the first time that people have claimed that the world is going to end.
Forget about whether your sandbox is a realistic enough test. There are even questions about how much safety you’re getting from a sandbox. So, we follow your advice, and put the AI in a box in order to test it. And then it escapes anyway, during the test.
That doesn’t seem like a reliable plan.
The idea that society is smart enough to build machine intelligence, but not smart enough to build a box to test it in does not seem credible to me:
Humans build boxes to put other humans in—and have a high success rate of keeping them inside when they put their minds to it. The few rogue agents that do escape are typically hunted down and imprisoned again. Basically the builders of the box are much stronger and more powerful than what it will contain. Machine intelligence testing seems unlikely to be significantly different from that situation.
The cited “box” scenario discusses the case of weak gatekeepers and powerful escapees. That scenario isn’t very relevant in this case—since we will have smart machines on both sides when restraining intelligent machines in order to test them.
Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.
DOOM is actually big business. Check out all the disaster movies out there. DOOM sells. What could be more important than… THE END OF THE WORLD? What greater cause could there be than… SAVING THE WORLD? So, people buy the DOOM merchandise, contribute the DOOM dollars, and warn their friends about the impending DOOM—thus perpetuating the DOOM virus. That is part of why there have been so many DOOM prophets—DOOM pays.
The more I think about this, the more it seems incorrect.
Human brains are not particularly friendly.
I would disagree. The overwhelming majority of the average human’s life is spent peacefully. It is actually fairly remarkable how rarely we have significant conflict, especially considering the relatively overcrowded places that humans live. Not to mention that it is only a small proportion of the human population that engages other humans destructively (not by proxy).
The overwhelming majority of the average human’s life is also spent in conditions where they are on relatively even grounds with everyone else. But once you start looking at what happens when people end up in situations where they are clearly more powerful than others? And can treat those others the way they like and without fear of retribution? Ugly.
I disagree. While there are some spectacular examples of what you describe, and they are indeed ugly, by and large there is wide distribution of hierarchical disparity even in daily life which is more often than not mutually beneficent.
As an emperor I optimize my empire by ensuring that my subjects are philosophically and physically satisfied do I not? I think there is plenty of evidence to support this philosophy as the most sustainable (and positive) of hierarchical models; after all some of the most successful businesses are laterally organized.
A certain philosophy being the most sustainable and positive isn’t automatically the same as being the one people tend to adopt. Plus the answer to your question depends on what you’re trying to optimize.
Also, it sounds like you’re still talking about a situation where people don’t actually have ultimate power. If we’re discussing a potential hard takeoff scenario, then considerations such as “which models have been the most successful for businesses before” don’t really apply. Any entity genuinely undergoing a hard takeoff is one that isn’t afterwards bound by what’s successful for humans, any more than we are bound by the practices that work the best for ants.
I think there is more than ample evidence to suggest that those are significantly less likely to be adopted—however wouldn’t a group of people who know that and can correct for it be the best test case of implementing an optimized strategy?
I hold the view that it is unnecessary to hold ultimate power over FAI. I certainly wouldn’t bind it to what has worked for humans thus far. Don’t fear the AI, find a way to assimilate.