3. Meta-ethics. It’s hard to be rational if you don’t know where your values are supposed to come from.
4. Normative ethics. How much weight to put on altruism? Population ethics. Hedonic vs preference utilitarianism. Moral circle. Etc. It’s hard to be rational if you don’t know what your values are.
5. Which mental subsystem has one’s real values, or how to weigh them.
6. How to handle moral uncertainty? For example should we discount total utilitarianism because we would have made a deal to for total utilitarianism to give up control in this universe?
7. If we apply UDT to humans, what does it actually say in various real-life situations like voting or contributing to x-risk reduction?
8. Does Aumann Agreement apply to humans, and if so how?
9. Meta-philosophy. It’s hard to be rational if one doesn’t know how to solve philosophical problems related to rationality.
10. It’s not clear how selfishness works in UDT, which might be a problem if that’s the right decision theory for humans.
11. Bargaining, politics, building alliances, fair division, we still don’t know how to apply game theory to a lot of messy real-world problems, especially those involving more than a few people.
12. Reality fluid vs. caring measure. Subjective anticipation. Anthropics in general.
13. What is the nature of rationality, and more generally normativity?
14. What is the right way to handle logical uncertainty, and how does that interact with decision theory, bargaining, and other problems?
Comparing the rate of problems opened vs problems closed, we have so far to go....
One more, because one of my posts presented two open problems, and I only listed one of them above:
15. Our current theoretical foundations for rationality all assume a fully specified utility function (or the equivalent), or at least a probability distribution on utility functions (to express moral/value uncertainty). But to the extent that humans can be considered to have a utility function at all, it’s may best be viewed as a partial function that returns “unknown” for most of the input domain. Our current decision theories can’t handle this because they would end up trying to add “unknown” to a numerical value during expected utility computation. Forcing humans to come up with an utility function or even a probability distribution on utility functions in order to use decision theory seems highly unsafe so we need an alternative.
If you just do it the straightforward way, any option you can choose would have a non-zero probability of producing an outcome with “unknown” or NaN utility. If multiply those two numbers together you get NaN, then if you add that to other probability*utility values as part of your expected utility computation you will end up with NaN as your final expected utility. I don’t see how to avoid this, hence my question.
Comparing the rate of problems opened vs problems closed, we have so far to go....
That’s always the case when investigating a new field, as you clarify the issues and get more specialised. It doesn’t mean necessarily that the majority of work remains ahead of us (though it doesn’t mean the opposite, either).
Oh, I see. My “we have so far to go” wasn’t meant to express optimism that we’re at the beginning of a burgeoning field (but I can see now how it could easily be interpreted that way). I meant more that when we’ve eventually got rationality fully figured out (i.e., some time after the Singularity), the amount of knowledge we have about it today will probably seem tiny in comparison.
Unless you have doubts that even superintelligence can answer my questions?
Given the way the answer feature goes, I think it would make more sense to have every single point as a separate answer to allow people to vote on them.
We have some vague plans to build something that’d make that process cleaner. I’m not sure if we’ll get to it soon, but meanwhile don’t think it’s that urgent to split the answers up.
I went through all my LW posts and gathered the ones that either presented or reminded me of some problem in human rationality.
1. As we become more rational, how do we translate/transfer our old values embodied in the less rational subsystems?
2. How to figure out one’s comparative advantage?
3. Meta-ethics. It’s hard to be rational if you don’t know where your values are supposed to come from.
4. Normative ethics. How much weight to put on altruism? Population ethics. Hedonic vs preference utilitarianism. Moral circle. Etc. It’s hard to be rational if you don’t know what your values are.
5. Which mental subsystem has one’s real values, or how to weigh them.
6. How to handle moral uncertainty? For example should we discount total utilitarianism because we would have made a deal to for total utilitarianism to give up control in this universe?
7. If we apply UDT to humans, what does it actually say in various real-life situations like voting or contributing to x-risk reduction?
8. Does Aumann Agreement apply to humans, and if so how?
9. Meta-philosophy. It’s hard to be rational if one doesn’t know how to solve philosophical problems related to rationality.
10. It’s not clear how selfishness works in UDT, which might be a problem if that’s the right decision theory for humans.
11. Bargaining, politics, building alliances, fair division, we still don’t know how to apply game theory to a lot of messy real-world problems, especially those involving more than a few people.
12. Reality fluid vs. caring measure. Subjective anticipation. Anthropics in general.
13. What is the nature of rationality, and more generally normativity?
14. What is the right way to handle logical uncertainty, and how does that interact with decision theory, bargaining, and other problems?
Comparing the rate of problems opened vs problems closed, we have so far to go....
One more, because one of my posts presented two open problems, and I only listed one of them above:
15. Our current theoretical foundations for rationality all assume a fully specified utility function (or the equivalent), or at least a probability distribution on utility functions (to express moral/value uncertainty). But to the extent that humans can be considered to have a utility function at all, it’s may best be viewed as a partial function that returns “unknown” for most of the input domain. Our current decision theories can’t handle this because they would end up trying to add “unknown” to a numerical value during expected utility computation. Forcing humans to come up with an utility function or even a probability distribution on utility functions in order to use decision theory seems highly unsafe so we need an alternative.
Does it help to propagate “unknown” through computations by treating it like NaN? Or would that tend to turn the answer to every question into NaN?
If you just do it the straightforward way, any option you can choose would have a non-zero probability of producing an outcome with “unknown” or NaN utility. If multiply those two numbers together you get NaN, then if you add that to other probability*utility values as part of your expected utility computation you will end up with NaN as your final expected utility. I don’t see how to avoid this, hence my question.
Thank you, this is great!
That’s always the case when investigating a new field, as you clarify the issues and get more specialised. It doesn’t mean necessarily that the majority of work remains ahead of us (though it doesn’t mean the opposite, either).
I’m confused what point you’re making. Isn’t P(the majority of work remains ahead of us | investigating a new field) quite high?
It’s hard to tell whether we’re at the beginning of a bugeonning field, or about to hit diminishing returns.
Oh, I see. My “we have so far to go” wasn’t meant to express optimism that we’re at the beginning of a burgeoning field (but I can see now how it could easily be interpreted that way). I meant more that when we’ve eventually got rationality fully figured out (i.e., some time after the Singularity), the amount of knowledge we have about it today will probably seem tiny in comparison.
Unless you have doubts that even superintelligence can answer my questions?
Given the way the answer feature goes, I think it would make more sense to have every single point as a separate answer to allow people to vote on them.
We have some vague plans to build something that’d make that process cleaner. I’m not sure if we’ll get to it soon, but meanwhile don’t think it’s that urgent to split the answers up.
That’d probably be messy.