Actually I’d be quite confident in no fates worse than death emerging from that scenario. There wouldn’t be time for anyone to mess up on constructing something almost in our moral frame of reference, just the first working version of AIXI / Schmidhuber’s Godel machine eating its future light cone.
Oh, come on. I’m sure at least a few people would end up with a Fate Worse Than Death instead! ;)
Actually I’d be quite confident in no fates worse than death emerging from that scenario. There wouldn’t be time for anyone to mess up on constructing something almost in our moral frame of reference, just the first working version of AIXI / Schmidhuber’s Godel machine eating its future light cone.
but it can’t AIXI the other halting oracles?