I am pretty sure, that as soon as first AGI’s arrive on the market, people would start to take possible dangers more seriously.
And it will be quite likely at that point that we are much closer to having an AGI that will foom than to having an AI that won’t kill us and that it is too late.
I know it is a local trope that death and destruction is apparent and necessary logical conclusion of creating an intelligent machine capable of self improvement and goal modification, but I certainly don’t share those sentiments.
How do you estimate the probability that AGI’s won’t take over the world (people who constructed them may use them for that purpose, but it is a different story), and would be used as simple tools and advisors in the same way boring, old fashioned and safe way 100% of our current technology is used?
I am explicitly saying that MRI or FAI are pointless, or anything like that. I just want to point out that they posture as if they were saving the world from imminent destruction, while it is no where certain weather said danger is really the case.
How do you estimate the probability that AGI’s won’t take over the world (people who constructed them may use them for that purpose, but it is a different story), and would be used as simple tools and advisors in the same way boring, old fashioned and safe way 100% of our current technology is used?
1%? I believe that it is nearly impossible to use a foomed AI in a safe manner without explicitly trying to do so. That’s kind of why I am worried about the threat of any uFAI developed before it is proven that we can develop a Friendly one and without using whatever the proof entails.
Anyway,
...would be used as simple tools and advisors in the same way boring, old fashioned and safe way 100% of our current technology is used?
I wasn’t aware that we use a 100% of our current technology in a safe way.
You may have a different picture of current technology than I do, or you may be extrapolating different aspects. We’re already letting software optimize the external world directly, with slightly worrying results. You don’t get from here to strictly and consistently limited Oracle AI without someone screaming loudly about risks. In addition, Oracle AI has its own problems (tell me if the LW search function doesn’t make this clear).
Some critics appear to argue that the direction of current tech will automatically produce CEV. But today’s programs aim to maximize a behavior, such as disgorging money. I don’t know in detail how Google filters its search results, but I suspect they want to make you feel more comfortable with links they show you, thus increasing clicks or purchases from sometimes unusually dishonest ads. They don’t try to give you whatever information a smarter, better informed you would want your current self to have. Extrapolating today’s Google far enough doesn’t give you a Friendly AI, it gives you the making of a textbook dystopia.
Oh yes, I know that. It just says a lot that our best shot is still decades away from achieving it’s goal.
Which, to be fair, isn’t saying much.
Seeing as we are talking about speculative dangers coming from a speculative technology that has yet to be developed, it seems pretty understandable.
I am pretty sure, that as soon as first AGI’s arrive on the market, people would start to take possible dangers more seriously.
And it will be quite likely at that point that we are much closer to having an AGI that will foom than to having an AI that won’t kill us and that it is too late.
I know it is a local trope that death and destruction is apparent and necessary logical conclusion of creating an intelligent machine capable of self improvement and goal modification, but I certainly don’t share those sentiments.
How do you estimate the probability that AGI’s won’t take over the world (people who constructed them may use them for that purpose, but it is a different story), and would be used as simple tools and advisors in the same way boring, old fashioned and safe way 100% of our current technology is used?
I am explicitly saying that MRI or FAI are pointless, or anything like that. I just want to point out that they posture as if they were saving the world from imminent destruction, while it is no where certain weather said danger is really the case.
1%? I believe that it is nearly impossible to use a foomed AI in a safe manner without explicitly trying to do so. That’s kind of why I am worried about the threat of any uFAI developed before it is proven that we can develop a Friendly one and without using whatever the proof entails.
Anyway,
I wasn’t aware that we use a 100% of our current technology in a safe way.
You may have a different picture of current technology than I do, or you may be extrapolating different aspects. We’re already letting software optimize the external world directly, with slightly worrying results. You don’t get from here to strictly and consistently limited Oracle AI without someone screaming loudly about risks. In addition, Oracle AI has its own problems (tell me if the LW search function doesn’t make this clear).
Some critics appear to argue that the direction of current tech will automatically produce CEV. But today’s programs aim to maximize a behavior, such as disgorging money. I don’t know in detail how Google filters its search results, but I suspect they want to make you feel more comfortable with links they show you, thus increasing clicks or purchases from sometimes unusually dishonest ads. They don’t try to give you whatever information a smarter, better informed you would want your current self to have. Extrapolating today’s Google far enough doesn’t give you a Friendly AI, it gives you the making of a textbook dystopia.