I am not at all sure why the humans wouldn’t just turn the AI on again anyway if it were only 99% probable.
Anyway, this reminds me of an oracle system I devised for a fantasy story I never got around to writing—The oracle doesn’t always respond, and if they do respond, they tell you what would have happened if they hadn’t responded. One of the rules I quickly had to make for the Oracle was that if they didn’t say anything, you didn’t get to ask again.
I thought (at the time, some time ago) that the Oracle, seeking to be most helpful, would soon converge on answering only around 2⁄3 − 4⁄5 of the time so that people wouldn’t go and do stupid things in response to the extreme upset of not getting an answer.
I am not at all sure why the humans wouldn’t just turn the AI on again anyway if it were only 99% probable.
Anyway, this reminds me of an oracle system I devised for a fantasy story I never got around to writing—The oracle doesn’t always respond, and if they do respond, they tell you what would have happened if they hadn’t responded. One of the rules I quickly had to make for the Oracle was that if they didn’t say anything, you didn’t get to ask again.
I thought (at the time, some time ago) that the Oracle, seeking to be most helpful, would soon converge on answering only around 2⁄3 − 4⁄5 of the time so that people wouldn’t go and do stupid things in response to the extreme upset of not getting an answer.
That’s a human institution problem, that seems more solvable (at least, we shouldn’t run the AI if it isn’t solved).