“The idea that superintelligences will more closely approximate rational utilitarian agents than current organisms is based on the idea that they will be more rational, suffer from fewer resource constraints, and be less prone to problems that cause them to pointlessly burn through their own resources.”
But this in my book is not a firm basis. A firm basis requires a theory of AI. Only then can you talk about whether they will pointlessly burn through resources or not.
There are lots of things that are not obviously pointlessly burning through resources, but might still be doing so. Things like trying to prove P == NP if this happens to be improvable either way, modelling the future of the earth’s climate without being able to take into account new anthropogenic influences (such as the change in albedo of solar cells, arcologies, or even unfolding nanoblooms).
Even spending time on this blog might be burning brain and computer cycles, perhaps we should be laying down our thoughts we want to survive in stone, in case we bomb ourselves back into the stone age.
“The idea that superintelligences will more closely approximate rational utilitarian agents than current organisms is based on the idea that they will be more rational, suffer from fewer resource constraints, and be less prone to problems that cause them to pointlessly burn through their own resources.”
But this in my book is not a firm basis. A firm basis requires a theory of AI. Only then can you talk about whether they will pointlessly burn through resources or not.
There are lots of things that are not obviously pointlessly burning through resources, but might still be doing so. Things like trying to prove P == NP if this happens to be improvable either way, modelling the future of the earth’s climate without being able to take into account new anthropogenic influences (such as the change in albedo of solar cells, arcologies, or even unfolding nanoblooms).
Even spending time on this blog might be burning brain and computer cycles, perhaps we should be laying down our thoughts we want to survive in stone, in case we bomb ourselves back into the stone age.