The TL;DR seems to be: “We only need a lower bound on the catastrophe/reasonable impact ratio, and an idea about how much utility is available for reasonable plans.”
This seems good… can you confirm my understanding below is correct?
2) RE: “How much utility is available”: I guess we can just set a targeted level of utility gain, and it won’t matter if there are plans we’d consider reasonable that would exceed that level? (e.g. “I’d be happy if we can make 50% more paperclips at the same cost in the next year.”)
1) RE: “A lower bound”: this seems good because we don’t need to know how extreme catastrophes could be, we can just say: “If (e.g.) the earth or the human species ceased to exist as we know it within the year, that would be catastrophic”.
OK, thanks.
The TL;DR seems to be: “We only need a lower bound on the catastrophe/reasonable impact ratio, and an idea about how much utility is available for reasonable plans.”
This seems good… can you confirm my understanding below is correct?
2) RE: “How much utility is available”: I guess we can just set a targeted level of utility gain, and it won’t matter if there are plans we’d consider reasonable that would exceed that level? (e.g. “I’d be happy if we can make 50% more paperclips at the same cost in the next year.”)
1) RE: “A lower bound”: this seems good because we don’t need to know how extreme catastrophes could be, we can just say: “If (e.g.) the earth or the human species ceased to exist as we know it within the year, that would be catastrophic”.
That’s correct.