I mean, would you say that the AGI I’m building in my basement is “Friendly” just because I expect it to do good things, even though it also might paperclip the universe 2% of the time?
If that is a once off 2% chance of failure then I’ll go with it and call it Friendly Enough For Me. Come to think of it I may be tempted by 2% per 1,000 years.
Come to think of it I may be tempted by 2% per 1,000 years.
I’m virtually certain that you realize the implications; for instance, you’re saying you’re tempted by a 50% chance of paperclipping per 34,000 years. I’m less clear on how you could justify being tempted.
I’m virtually certain that you realize the implications; for instance, you’re saying you’re tempted by a 50% chance of paperclipping per 34,000 years. I’m less clear on how you could justify being tempted.
Start with assigning a very low probability on something better occurring. Then discount somewhat the extremely good options where you live billions of years, not valuing years on a linear scale. Then consider what you can do in 10,000 years with a super-intelligence backing you up. For example, it could build you a relativistic rocket and send you off fast enough that you are outside a future paper clipper’s future light cone. Possibly sending multiple copies of the human race out in various directions and with various planned durations of flight (given that you don’t know when Mostly Friendly is going to go nuts).
I haven’t done any maths on what the figures would need to be for me to actually choose that scenario. That’s why I say ‘may’. It is certainly worth considering seriously.
If that is a once off 2% chance of failure then I’ll go with it and call it Friendly Enough For Me. Come to think of it I may be tempted by 2% per 1,000 years.
I’m virtually certain that you realize the implications; for instance, you’re saying you’re tempted by a 50% chance of paperclipping per 34,000 years. I’m less clear on how you could justify being tempted.
Start with assigning a very low probability on something better occurring. Then discount somewhat the extremely good options where you live billions of years, not valuing years on a linear scale. Then consider what you can do in 10,000 years with a super-intelligence backing you up. For example, it could build you a relativistic rocket and send you off fast enough that you are outside a future paper clipper’s future light cone. Possibly sending multiple copies of the human race out in various directions and with various planned durations of flight (given that you don’t know when Mostly Friendly is going to go nuts).
I haven’t done any maths on what the figures would need to be for me to actually choose that scenario. That’s why I say ‘may’. It is certainly worth considering seriously.
Good answer.