It’s not just how much space there is, it’s also how easy it is to reach. I think there is a lot of space, but that it requires a lot of computing power to reach. If a system is merely superhuman, then destroying humankind will not be a good strategy. A system would have to climb quite high before cooperation with humans started to look suboptimal. (There are also Mark Waser’s arguments that cooperation looks more and more optimal as a system gets smarter, but he hasn’t established that mathematically...)
I understand your point. We agree that there is a lot of space above, we don’t agree how easy it is to climb up there.
With a LOT of CPU, it would be almost trivial. But what can we do (what can AI do) with much less?
We have done many amazing things in mathematics mostly at the state of the CPU starvation. We have accomplished a lot with a very limited calculation resources. So much so, that we assumed how “thinking is not equal calculating”.
A substantial CPU usage optimization may be possible for an AI and by an AI. I guess.
One question which is pertinent here: is there anything dramatically better than Levin search for general problem solving? (That’s a real question, not intended rhetorically.)
Some approximate method could be as nearly as good and a lot cheaper to compute. I am at least 60% confident it exists.
Some highly optimized implementation might be acceptable. Only 30% confident that there is one such.
For the case, that there is something different and radically better, I give 25%.
And there is another optic. Say, that I have 10^20 CPU cycles every second. How much of a Levin search I can afford with this? Maybe it is enough. I give 80% chance for this one.
What if I have only 10^15 cps and still enough? I give 50% chance for this.
And maybe even 10^10 cps is quite all right. So that I could do it today, had been smarter. I give it 10%.
Of course, those are all wild guesses, but it’s my best answer I can give.
A lot can be boiled down to the question—How much space is above our intelligence level?
If there is a plenty of intelligence space above, then the problem can be huge. In that case the AI can surprise us anytime.
If there is not, it is not a problem and the AI will not be very much smarter than we are.
I think that there IS a plenty of room up there.
It’s not just how much space there is, it’s also how easy it is to reach. I think there is a lot of space, but that it requires a lot of computing power to reach. If a system is merely superhuman, then destroying humankind will not be a good strategy. A system would have to climb quite high before cooperation with humans started to look suboptimal. (There are also Mark Waser’s arguments that cooperation looks more and more optimal as a system gets smarter, but he hasn’t established that mathematically...)
I understand your point. We agree that there is a lot of space above, we don’t agree how easy it is to climb up there.
With a LOT of CPU, it would be almost trivial. But what can we do (what can AI do) with much less?
We have done many amazing things in mathematics mostly at the state of the CPU starvation. We have accomplished a lot with a very limited calculation resources. So much so, that we assumed how “thinking is not equal calculating”.
A substantial CPU usage optimization may be possible for an AI and by an AI. I guess.
One question which is pertinent here: is there anything dramatically better than Levin search for general problem solving? (That’s a real question, not intended rhetorically.)
Some approximate method could be as nearly as good and a lot cheaper to compute. I am at least 60% confident it exists.
Some highly optimized implementation might be acceptable. Only 30% confident that there is one such.
For the case, that there is something different and radically better, I give 25%.
And there is another optic. Say, that I have 10^20 CPU cycles every second. How much of a Levin search I can afford with this? Maybe it is enough. I give 80% chance for this one.
What if I have only 10^15 cps and still enough? I give 50% chance for this.
And maybe even 10^10 cps is quite all right. So that I could do it today, had been smarter. I give it 10%.
Of course, those are all wild guesses, but it’s my best answer I can give.