Right now no one knows how to maximize either paper clips or alpine villages. The first thing we know how to do will probably be some poorly-understood recursively self-improving cycle of computer code interacting with other computer code. Then the resulting intelligence will start converging on some goal and converge on capabilities to optimize it extremely powerfully. The problem is that that emergent goal will be a lot more random and arbitrary than an alpine village. Most random things that this process can land on look like a paper clip in how devoid of human value they are, not like an alpine village which has a very significant amount of human value in it.
I know, that “Right now no one knows how to maximize either paper clips …”. I know. But paper clips have been the official currency of these debates for almost 20 years now. Suddenly they aren’t, just because “right now no one knows how to”?
And then, you are telling me what is to be done first and how?
Yes it’s an important insight that paper clips are a representative example of a much bigger and simpler space of optimization targets than alpine villages.
Sure, but “alpine villages” or something alike, were called “astronomical waste” in the MIRI’s language from the old days. When the “fun space”, as they called it, was nearly infinite. Now they say, its volume is almost certainly zero.
Right now no one knows how to maximize either paper clips or alpine villages. The first thing we know how to do will probably be some poorly-understood recursively self-improving cycle of computer code interacting with other computer code. Then the resulting intelligence will start converging on some goal and converge on capabilities to optimize it extremely powerfully. The problem is that that emergent goal will be a lot more random and arbitrary than an alpine village. Most random things that this process can land on look like a paper clip in how devoid of human value they are, not like an alpine village which has a very significant amount of human value in it.
I know, that “Right now no one knows how to maximize either paper clips …”. I know. But paper clips have been the official currency of these debates for almost 20 years now. Suddenly they aren’t, just because “right now no one knows how to”?
And then, you are telling me what is to be done first and how?
Yes it’s an important insight that paper clips are a representative example of a much bigger and simpler space of optimization targets than alpine villages.
Sure, but “alpine villages” or something alike, were called “astronomical waste” in the MIRI’s language from the old days. When the “fun space”, as they called it, was nearly infinite. Now they say, its volume is almost certainly zero.