Sure, in case of erasure you can decide to use oracles less, and compensate your clients with money you got from “erasure insurance” (since that’s a low probability event). But that doesn’t seem to solve the problem I’m talking about—UFAI arising naturally in erasure-worlds and spreading to non-erasure-worlds through oracles.
The problem you were talking about seemed to rely on bucket brigades. I agree that UFAIs jumping back a single step is a fair concern. (Though I guess you could counterfactually have enough power to halt AGI research completely...) I’m trying to address it elsethread. :)
Ah, sorry, you’re right. To prevent bucket brigades, it’s enough to stop using oracles for N days whenever an N-day oracle has an erasure event, and the money from “erasure insurance” can help with that. When there are no erasure events, we can use oracles as often as we want. That’s a big improvement, thanks!
Very worthwhile concern, and I will think about it more.
In case of erasure, you should be able to get enough power to prevent another UFAI summoning session.
Sure, in case of erasure you can decide to use oracles less, and compensate your clients with money you got from “erasure insurance” (since that’s a low probability event). But that doesn’t seem to solve the problem I’m talking about—UFAI arising naturally in erasure-worlds and spreading to non-erasure-worlds through oracles.
The problem you were talking about seemed to rely on bucket brigades. I agree that UFAIs jumping back a single step is a fair concern. (Though I guess you could counterfactually have enough power to halt AGI research completely...) I’m trying to address it elsethread. :)
Ah, sorry, you’re right. To prevent bucket brigades, it’s enough to stop using oracles for N days whenever an N-day oracle has an erasure event, and the money from “erasure insurance” can help with that. When there are no erasure events, we can use oracles as often as we want. That’s a big improvement, thanks!
Good idea.