But I guess I’m sufficiently confident in “>50% chance that it’s destructive” that I’ll argue for that.
Fwiw 50% on doom in the story I told seems plausible to me; maybe I’m at 30% but that’s very unstable. I don’t think we disagree all that much here.
Then we can start talking about capability windows etc., but I don’t think that was your objection here.
Capability windows are totally part of the objection. If you completely ignore capability windows / compute restrictions then you just run AIXI (or AIXI-tl if you don’t want something uncomputable) and die immediately.
Fwiw 50% on doom in the story I told seems plausible to me; maybe I’m at 30% but that’s very unstable. I don’t think we disagree all that much here.
Capability windows are totally part of the objection. If you completely ignore capability windows / compute restrictions then you just run AIXI (or AIXI-tl if you don’t want something uncomputable) and die immediately.