Is the xkcd rock-placing man in any danger if he creates a UFAI? Apparently not, since he is, to quote Hawking, the one who “breathes fire into the equations”. Is creating an AGI of use to him? Probably, if he has questions it can answer for him (by assumption, he just knows the basic rules of stone-laying, not everything there is to know). Can there be a similar way to protect actual humans from a potential rogue AI?
Assuming the premises of the situation, yes to your first question:
He may be argued into something that is not in his interest by the UFAI. (On the other hand, Rock-Placing Man evidently does not have a standard mental and physical architecture, so maybe he also happens to be immune to such attacks.)
The UFAI may take over his simulated universe and turn it into simulated paperclips.
Either the rock-placing man is running the AI so slowly that it’s not useful for anything or he runs the risk of falling prey to considerations that have already been discussed on LW surrounding oracle AI.
Steelmanning is probably a good thing to do (and I’m not good at doing it), but I think it’s bad form to ask that somebody steelman you.
Either the rock-placing man is running the AI so slowly that it ’s not useful for anything or he runs the risk of falling prey to considerations that have already been discussed on LW surrounding oracle AI.
This would be a useful conjecture if you can formalize it, or maybe a theorem if you can prove it.
What is with LW people and theorems? The situation you’ve described is nowhere near formalized enough for there to be anything reasonable to say about it at the level of precision and formality that warrants a word like “theorem.”
Is the xkcd rock-placing man in any danger if he creates a UFAI? Apparently not, since he is, to quote Hawking, the one who “breathes fire into the equations”. Is creating an AGI of use to him? Probably, if he has questions it can answer for him (by assumption, he just knows the basic rules of stone-laying, not everything there is to know). Can there be a similar way to protect actual humans from a potential rogue AI?
Assuming the premises of the situation, yes to your first question:
He may be argued into something that is not in his interest by the UFAI. (On the other hand, Rock-Placing Man evidently does not have a standard mental and physical architecture, so maybe he also happens to be immune to such attacks.)
The UFAI may take over his simulated universe and turn it into simulated paperclips.
True, but it’s easy to deal with, just place one row of rocks differently and wipe the UFAI bugger out. Humans would not have this luxury.
You mean run them so slowly that they’re not useful for anything?
Ever heard of steelmanning?
Either the rock-placing man is running the AI so slowly that it’s not useful for anything or he runs the risk of falling prey to considerations that have already been discussed on LW surrounding oracle AI.
Steelmanning is probably a good thing to do (and I’m not good at doing it), but I think it’s bad form to ask that somebody steelman you.
This would be a useful conjecture if you can formalize it, or maybe a theorem if you can prove it.
What is with LW people and theorems? The situation you’ve described is nowhere near formalized enough for there to be anything reasonable to say about it at the level of precision and formality that warrants a word like “theorem.”