This doesn’t sound like an argument Yudkowsky would make, though it seems to have some similar concepts. And it’s interesting food for thought regardless—thanks! Looking forward to the rest of the series.
This doesn’t sound like an argument Yudkowsky would make
Yeah, I can’t immediately find the link but I recall that Eliezer had a tweet in the past few months along the lines of: If ASI wants to tile the universe with one thing, then it wipes out humanity. If ASI wants to tile the universe with sixteen things , then it also wipes out humanity.
My mental-model-of-Yudkowsky would bring up “tiny molecular squiggles” in particular for reasons a bit more analogous to the CoastRunners behavior (video)—if any one part of the motivational system is (what OP calls) decomposable etc., then the ASI would find the “best solution” to maximizing that part. And if numbers matter, then the “best solution” would presumably be many copies of some microscopic thing.
I don’t argue at any point that ASIs will have a single goal. The argument goes through equally well if it has many. The question is why some of those goals are of the form “tile the universe with squiggles” at all. That’s the part I’m addressing in this post.
This doesn’t sound like an argument Yudkowsky would make, though it seems to have some similar concepts. And it’s interesting food for thought regardless—thanks! Looking forward to the rest of the series.
Yeah, I can’t immediately find the link but I recall that Eliezer had a tweet in the past few months along the lines of: If ASI wants to tile the universe with one thing, then it wipes out humanity. If ASI wants to tile the universe with sixteen things , then it also wipes out humanity.
My mental-model-of-Yudkowsky would bring up “tiny molecular squiggles” in particular for reasons a bit more analogous to the CoastRunners behavior (video)—if any one part of the motivational system is (what OP calls) decomposable etc., then the ASI would find the “best solution” to maximizing that part. And if numbers matter, then the “best solution” would presumably be many copies of some microscopic thing.
I don’t argue at any point that ASIs will have a single goal. The argument goes through equally well if it has many. The question is why some of those goals are of the form “tile the universe with squiggles” at all. That’s the part I’m addressing in this post.