The problem is getting an Oracle to answer useful questions.
Paperclip Manufacturer: “How do I make paperclips?”
Oracle: Shows him designs for a paperclip maximizer
Paperclip Manufacturer: “How do I make paperclips, in a way I’d actually be willing to do?”
Oracle: Shows him designs for an innocuous-looking paperclip maximizer
Once you get it to answer your question without designing an X-maximiser, you’ve pretty much solved FAI.
The problem is getting an Oracle to answer useful questions.
Paperclip Manufacturer: “How do I make paperclips?”
Oracle: Shows him designs for a paperclip maximizer
Paperclip Manufacturer: “How do I make paperclips, in a way I’d actually be willing to do?”
Oracle: Shows him designs for an innocuous-looking paperclip maximizer
Once you get it to answer your question without designing an X-maximiser, you’ve pretty much solved FAI.