Origin: I believe this was inspired by something Wei Dai said, but it is possible the originator is a simulation of him inside my head. I developed it after a brief conversation with Derek Shiller. Any errors are my own.
Epistemic status: seems like an interesting idea.
Terminology: in this post we will refer to moral theories agreeing to different levels of influence on agents, and agents as having different levels of impact in the world.
Moral Trade
One approach to moral uncertainty is to have different moral theories negotiate behind a veil of ignorance, Harsanyi style. There are a number of different approaches you could take here, like Moral Parliaments. What we care about here is the potential for these moral theories to do mutually beneficial (by each’s own lights) trades with each other.
A particular trade they might want to do would be to exchange influence across different agents and possible worlds based on their ability to exert counterfactual impact, since some theories assign higher value to being able to influence a lot of impact.
Impact Distribution
You might assign some credence to deontological moral theories, where the most important thing is that you personally behave well and do not violate any moral constraints—a naturally bounded quantity for each individual. You might also have some credence in an aggregative moral theory, where all that matters is maximising some global summation, like traditional hedonic utilitarianism, which has a much longer tailed distribution of impact.
Suppose some people are much more influential than others, and behind the veil of ignorance, the negotiating moral theories were uncertain as to how influential you would be. They might then agree to a trade where the deontological moral theory gains significant influence over you if you turn out to be a typical person, while the aggregative moral theory gains influence in the small fraction of worlds where you become an important leader.
For example, if real-world aggregate impact (which the aggregative theory cares about) was log-normally distributed, but each individual has the ability to personally sin or not (what the deontologist cares about), then the 75% of least influential individuals are 75% of the value for the deontologist, but only about 38% of the expected aggregated impact. So it might make sense for the deontologist to completely give up on the top 25% if they got the bottom 75% in return, and for the aggregative theory to accept this deal.
What might this look like? Well, perhaps a world in which ordinary people mainly follow moral rules like don’t lie and don’t steal, while leaders employ a bunch of economists and maximize total welfare, paying scant regard to their personal moral violations.
Parochialism, Indexicals and Cooperation
Above I presented a dichotomy of deontology vs uniform aggregation, but this can be extended to other theories. What matters is how fast a moral theory is satisfied as a function of the impact of each agent. The faster a theory is satisfied, the more it will prefer influence over a large number of less important people.
We could, with some abuse of notation, express the two theories above as:
Deontology trades to maximise ∑(agents)
Aggregative Consequentialism trades to maximise ∑(agents×impact)
There are theories in between these two. A simple example, noting the former can be re-written as ∑(agents×impact^0) is:
∑(agents×impact^(½))
One example could be group-specific theories, which care about aggregative impact but weigh it based on the group affected, like Nationalism. These theories prioritize influence in worlds where they significantly impact their favored group and less so in other worlds.
If these theories negotiate, they might agree to an indexical compromise. The British Nationalist theory cares mainly about worlds in which it is very impactful over Britain, and the Korean Nationalist theory cares equivalently Korean impact worlds, so behind the veil they agree that, if they find themselves in a good position to become an UK MP they’ll be an British Nationalist and likewise for Korea. This essentially ends up being an indexical theory.
One nice(?) aspect of this is it somewhat encourages cooperation, as it gives people a reason to perform well at their assigned roles. For example, moral theories emphasizing accounting integrity care deeply about the behavior of CFOs. Under this moral compromise, CFOs would more highly prioritize the integrity of their books and records.
Large and Small Worlds
Some worlds are larger than others. Behind the veil of ignorance, before we know any physics, we do not know how large the world will turn out to be.
Just as aggregative theories value influence over highly impactful agents more, they also prioritize influence over larger worlds. Therefore, they should be willing to relinquish influence over smaller worlds to secure influence over larger ones.
In particular, behind the veil—knowing logical and metaphysical facts but not physics—we might not know whether the accessible universe is infinite. If so, aggregative theories might be willing to give up all their influence over finite worlds to control the infinite ones.
This would imply that, having observed our world (or at least, our ability to impact it) is finite (and even kind of small, it seems to me), if we assigned any credence to non-aggregative theories then we should now reduce our credence / the influence of aggregative consequentialism. Whether or not the aggregate consequentialist theories give up all their influence would depend on the balance of bargaining power behind the veil—e.g. the difference between the weight assigned to aggregative theories vs the pre-physics prior assigned to an influenceable infinite universe.
Some other forms of consequentialism might survive—e.g. if you discount events far away in space or time, the finiteness of the world is potentially not a huge deal, so those theories likely retain some influence. But overall we shift towards less aggregative theories.
Is this fanatical, and ‘unfair’ to the aggregative theories? Maybe. But the aggregative theories really want to be able to make this trade. If there is any chance of infinite influence, that is ~all they care about. If anything I think the reverse is more plausible, that the fanaticism might be more ‘unfair’ to non-aggregative theories if they end up losing all influence in infinite worlds, because the ‘shut up and multiply’ argument that led us here seems more congenial to aggregative theories.
Alternatively, perhaps infinite worlds are somehow logically impossible, or infinite paralysis issues will kick in and make buying influence in those worlds unattractive to aggregative theories.
It’s also possible there are ethical systems that care even more about the largest worlds than aggregative ones—for example, if they use an exponential function. If you assign significant credence to them, they might dominate the largest worlds instead of the linear aggregative ones. Similarly, if there are possible worlds of many different infinite cardinalities, different theories might buy influence at different levels.
Auditing
When two parties make a trade whose resolution depends on future developments, like a cash-settled financial derivative or prediction market bet, it is important for the resolution criteria to be as clear and impartially evaluated as possible. Often parties are willing to incur significant overhead in order to ensure this, paying lawyers, auditors, clearing houses and exchanges to add robustness to the process.
Similarly, if moral theories made a trade that depended on the size of the universe, the speed of light, the viability of time travel, etc., you might imagine them also agreeing to dedicate a lot of resources to physics research in order to resolve which theory gets the influence.
Moral Trade, Impact Distributions and Large Worlds
Origin: I believe this was inspired by something Wei Dai said, but it is possible the originator is a simulation of him inside my head. I developed it after a brief conversation with Derek Shiller. Any errors are my own.
Epistemic status: seems like an interesting idea.
Terminology: in this post we will refer to moral theories agreeing to different levels of influence on agents, and agents as having different levels of impact in the world.
Moral Trade
One approach to moral uncertainty is to have different moral theories negotiate behind a veil of ignorance, Harsanyi style. There are a number of different approaches you could take here, like Moral Parliaments. What we care about here is the potential for these moral theories to do mutually beneficial (by each’s own lights) trades with each other.
A particular trade they might want to do would be to exchange influence across different agents and possible worlds based on their ability to exert counterfactual impact, since some theories assign higher value to being able to influence a lot of impact.
Impact Distribution
You might assign some credence to deontological moral theories, where the most important thing is that you personally behave well and do not violate any moral constraints—a naturally bounded quantity for each individual. You might also have some credence in an aggregative moral theory, where all that matters is maximising some global summation, like traditional hedonic utilitarianism, which has a much longer tailed distribution of impact.
Suppose some people are much more influential than others, and behind the veil of ignorance, the negotiating moral theories were uncertain as to how influential you would be. They might then agree to a trade where the deontological moral theory gains significant influence over you if you turn out to be a typical person, while the aggregative moral theory gains influence in the small fraction of worlds where you become an important leader.
For example, if real-world aggregate impact (which the aggregative theory cares about) was log-normally distributed, but each individual has the ability to personally sin or not (what the deontologist cares about), then the 75% of least influential individuals are 75% of the value for the deontologist, but only about 38% of the expected aggregated impact. So it might make sense for the deontologist to completely give up on the top 25% if they got the bottom 75% in return, and for the aggregative theory to accept this deal.
What might this look like? Well, perhaps a world in which ordinary people mainly follow moral rules like don’t lie and don’t steal, while leaders employ a bunch of economists and maximize total welfare, paying scant regard to their personal moral violations.
Parochialism, Indexicals and Cooperation
Above I presented a dichotomy of deontology vs uniform aggregation, but this can be extended to other theories. What matters is how fast a moral theory is satisfied as a function of the impact of each agent. The faster a theory is satisfied, the more it will prefer influence over a large number of less important people.
We could, with some abuse of notation, express the two theories above as:
Deontology trades to maximise ∑(agents)
Aggregative Consequentialism trades to maximise ∑(agents×impact)
There are theories in between these two. A simple example, noting the former can be re-written as ∑(agents×impact^0) is:
∑(agents×impact^(½))
One example could be group-specific theories, which care about aggregative impact but weigh it based on the group affected, like Nationalism. These theories prioritize influence in worlds where they significantly impact their favored group and less so in other worlds.
If these theories negotiate, they might agree to an indexical compromise. The British Nationalist theory cares mainly about worlds in which it is very impactful over Britain, and the Korean Nationalist theory cares equivalently Korean impact worlds, so behind the veil they agree that, if they find themselves in a good position to become an UK MP they’ll be an British Nationalist and likewise for Korea. This essentially ends up being an indexical theory.
One nice(?) aspect of this is it somewhat encourages cooperation, as it gives people a reason to perform well at their assigned roles. For example, moral theories emphasizing accounting integrity care deeply about the behavior of CFOs. Under this moral compromise, CFOs would more highly prioritize the integrity of their books and records.
Large and Small Worlds
Some worlds are larger than others. Behind the veil of ignorance, before we know any physics, we do not know how large the world will turn out to be.
Just as aggregative theories value influence over highly impactful agents more, they also prioritize influence over larger worlds. Therefore, they should be willing to relinquish influence over smaller worlds to secure influence over larger ones.
In particular, behind the veil—knowing logical and metaphysical facts but not physics—we might not know whether the accessible universe is infinite. If so, aggregative theories might be willing to give up all their influence over finite worlds to control the infinite ones.
This would imply that, having observed our world (or at least, our ability to impact it) is finite (and even kind of small, it seems to me), if we assigned any credence to non-aggregative theories then we should now reduce our credence / the influence of aggregative consequentialism. Whether or not the aggregate consequentialist theories give up all their influence would depend on the balance of bargaining power behind the veil—e.g. the difference between the weight assigned to aggregative theories vs the pre-physics prior assigned to an influenceable infinite universe.
Some other forms of consequentialism might survive—e.g. if you discount events far away in space or time, the finiteness of the world is potentially not a huge deal, so those theories likely retain some influence. But overall we shift towards less aggregative theories.
Is this fanatical, and ‘unfair’ to the aggregative theories? Maybe. But the aggregative theories really want to be able to make this trade. If there is any chance of infinite influence, that is ~all they care about. If anything I think the reverse is more plausible, that the fanaticism might be more ‘unfair’ to non-aggregative theories if they end up losing all influence in infinite worlds, because the ‘shut up and multiply’ argument that led us here seems more congenial to aggregative theories.
Alternatively, perhaps infinite worlds are somehow logically impossible, or infinite paralysis issues will kick in and make buying influence in those worlds unattractive to aggregative theories.
It’s also possible there are ethical systems that care even more about the largest worlds than aggregative ones—for example, if they use an exponential function. If you assign significant credence to them, they might dominate the largest worlds instead of the linear aggregative ones. Similarly, if there are possible worlds of many different infinite cardinalities, different theories might buy influence at different levels.
Auditing
When two parties make a trade whose resolution depends on future developments, like a cash-settled financial derivative or prediction market bet, it is important for the resolution criteria to be as clear and impartially evaluated as possible. Often parties are willing to incur significant overhead in order to ensure this, paying lawyers, auditors, clearing houses and exchanges to add robustness to the process.
Similarly, if moral theories made a trade that depended on the size of the universe, the speed of light, the viability of time travel, etc., you might imagine them also agreeing to dedicate a lot of resources to physics research in order to resolve which theory gets the influence.