Anyone smart enough to be dangerous is smart enough to be safe? I’m skeptical- folksy wisdom tells me that being smart doesn’t protect you from being stupid.
But in general, yes- the threat becomes more and more tangible as the barrier to AI gets lower and the number of players increases. At the moment, it seems pretty intangible, but I haven’t actually gone out and counted dangerously smart AI researchers- I might be surprised by how many there are.
To be clear, I was NOT trying to imply that we should actually right now form the Turing Police.
As I understand it, the argument (roughly) is that if you build an AI from scratch, using just tools available now, you will have to specify its utility function, in a way that the program can understand, as part of that process. Anyone actually trying to work out a utility function that can be programmed would have to have a fairly deep understanding—you can’t just type “make nice things happen and no bad things”, but have to think in terms that can be converted into C or Perl or whatever. In doing so, you would have to have some kind of understanding in your own head of what you’re telling the computer to do, and would be likely to avoid at least the most obvious failure modes.
However, in (say) twenty years that might not be the case—it might be (as an example) that we have natural language processing programs that can take a sentence like ‘make people happy’ and have some form of ‘understanding’ of it, while still not being Turing-test-passing, self-modification-capable fully general AIs. It could then get to the stage that some half-clever person could think “Hmm… If I put this and this and this together, I’ll have a self-modifying AI. And then I’ll just tell it to make everyone smile. What could go wrong?”
Anyone smart enough to be dangerous is smart enough to be safe? I’m skeptical- folksy wisdom tells me that being smart doesn’t protect you from being stupid.
But in general, yes- the threat becomes more and more tangible as the barrier to AI gets lower and the number of players increases. At the moment, it seems pretty intangible, but I haven’t actually gone out and counted dangerously smart AI researchers- I might be surprised by how many there are.
To be clear, I was NOT trying to imply that we should actually right now form the Turing Police.
As I understand it, the argument (roughly) is that if you build an AI from scratch, using just tools available now, you will have to specify its utility function, in a way that the program can understand, as part of that process. Anyone actually trying to work out a utility function that can be programmed would have to have a fairly deep understanding—you can’t just type “make nice things happen and no bad things”, but have to think in terms that can be converted into C or Perl or whatever. In doing so, you would have to have some kind of understanding in your own head of what you’re telling the computer to do, and would be likely to avoid at least the most obvious failure modes.
However, in (say) twenty years that might not be the case—it might be (as an example) that we have natural language processing programs that can take a sentence like ‘make people happy’ and have some form of ‘understanding’ of it, while still not being Turing-test-passing, self-modification-capable fully general AIs. It could then get to the stage that some half-clever person could think “Hmm… If I put this and this and this together, I’ll have a self-modifying AI. And then I’ll just tell it to make everyone smile. What could go wrong?”