The very first “compilation” I would suggest to your choice system would be to calculate the “Expected Utility of Success” for each Action.
1) It is rational to be prejudiced against Actions with a large difference between their “Expected Utility of Success” and their “Expected Utility”, even if that action might have the highest “Expected Utility”. People with a low tolerance for risk (constitutionally) would find the possible downside of such actions unacceptable.
2) Knowing the “Expected Utility of Success” gives information for future planning if success is realized. If success might be “winning a Hummer SUV in a raffle in December”, it would probably be irrational to construct a “too small” car port in November, even with success being non-certain.
Eliezer, I have a question.
In a simple model, how best to avoid the failure mode of taking a course of action with an unacceptable chance of leading to catastrophic failure? I am inclined to compute separately, for each action, its probability of leading to a catastrophic failure, and immediately exclude from further consideration those actions that cross a certain threshold.
The very first “compilation” I would suggest to your choice system would be to calculate the “Expected Utility of Success” for each Action.
1) It is rational to be prejudiced against Actions with a large difference between their “Expected Utility of Success” and their “Expected Utility”, even if that action might have the highest “Expected Utility”. People with a low tolerance for risk (constitutionally) would find the possible downside of such actions unacceptable.
2) Knowing the “Expected Utility of Success” gives information for future planning if success is realized. If success might be “winning a Hummer SUV in a raffle in December”, it would probably be irrational to construct a “too small” car port in November, even with success being non-certain.
Eliezer, I have a question.
In a simple model, how best to avoid the failure mode of taking a course of action with an unacceptable chance of leading to catastrophic failure? I am inclined to compute separately, for each action, its probability of leading to a catastrophic failure, and immediately exclude from further consideration those actions that cross a certain threshold.
Is this how you would proceed?