How do more sophisticated decision-makers differ from less sophisticated decision-makers in their behaviour and values?
Smarter more sophisticated decisionmakers engage in more and more complex commitments — including meta-commitments not to commit. Consequently, the values and behaviour of these more sophisticated decisionmakers “Greater Agents” are systematically biased compared to less sophisticated decisionmakers “Lesser Agents”.
*******************************
Compared to Lesser Agents, the Greater Agents are more judgemental, (self-)righteous, punish naivité, are more long-term oriented, adaptive, malleable,self-modifying, legibly trustworthy andpractice more virtue-signalling, strategic, engage in self-reflection & metacognition, engage in more thinking, less doing, symbolic reasoning, consistent & ‘rational’ in their preferences, they like money & currency more, sacred values less, value engagement in thinking over doing, engaged in more “global” conflicts [including multiverse-wide conflicts throguh acausal trade], less empirical, more rational, more universalistic in their morals, and more cosmopolitan in their esthetics, they are less likely to be threatened, andmore willing to martyr themselves, they willing toadmit their values’ origins and willing to barter on their values, engage in less frequent but more lethal war, love formal protocols and practice cryptographic magika.
Greater Agents punish Dumb Doves; Greater Agents are Judgemental.
Counterintuitively this means Greater Agent instances might fight to the death even if there is (seemingly) nothing to gain in this world. I.e. they might cultivate aBerserker ethos.
Greater Agents one-box on Newcomb’s problem
Greater Agents use some variant of LessWrong decision theories like UDT, FDT, TDT, LDT etc that recommended the ‘correct’ decision in situations where agents are able to engage in sophisticated commitments and simulate other agents with high fidelity.
Greater Agents are Better at Thinking and Meta-Thinking: more malleable, fluid, self-reflective, self-modifying.
As meta-cognitive agents can outperform less cognitive agents in certain environments Meta-Cognitive Agents will have a bias for thinking (and meta-thinking) over doing.
Trapped cognition Future agents will be more aware of traps in the POMDP of Thought.
Rely more on symbolic, logical, & formal reasoning in place of empirical and intuitive judgements.
Greater Agents are more able to control and harness the environment over long-time horizons; consequently, they have lower time preference and are more long-term oriented.
Greater Agents’ cognition is more ‘adaptive, malleable, fluid’ and they Self-Modify to be more Legibly Trustworthy
Sublety: an upcoming result by Nisan Stiennon says that a class of decisionmakers can (super-rationally) cooperate on the prisoners dilemma if (i) the Lipshitz coefficients of meta-beliefs—i.e. ‘I know that you know that I know that’- satisfy some constaints. Sophisticated decision makers may want to self-modify to constrain their ‘meta-adaptiveness’ (i.e. Lipshitz coefficients).
Greater Agents Agree and Bargain OR have Hansonian Origin Disputes
Aumann Agreement - rational honest reasoners can never ‘agree to disagree’. They have to Aumann Agree
Real agents aren’t perfect Bayesians. When the true hypothesis is not realizable OR one of the parties has access to privileged indexical information OR the beliefs of the agents are only an approximation of the Bayesian posterior. However, plausibly weaker forms of the Aumann Agreement Theorem may still be applicable.
OR have a value Dispute [Less Frequent more Lethal War]
War is a bargaining failure. Greater Agent employ more sophisticated bargaining solutions. If bargaining fails, wars are more lethal and rapid. They are more lethal for the same reason that forest fires are more rare but lethal when suppressed.
Origin Disputes may be resolved. For humans this would shake out to more explicitly incorporating evolutionary psychology in one’s moral and explicit reasoning, e.g. admitting and bargaining under the understanding of less-than-perfectly-prosocial evolutionary drives. For synthetically created organisms like AIs instead of evolutionary drives the origin.
OR they may fail to resolve by design. Vanessa Kosoy’s preferred alignment solution is a form of daemonic exorcism that explicitly biases the prior on the prior-origin to rule out exotic problems like Christiano’s Acausal Simulation Attack.
Greater Agents Love and Employ Contracts, Formal Protocols, and Cryptographic Magika
Or perhaps not. Arithmetic Rationality as a resource? Arithmetic rationality may be better thought of as a resource. Agent-egregores that are more able to bargain across different branches of the multiverse will—i.e. are more able to sacrifice minor loss in one branch for major gain in another brach—collect into more arithmetically rationally superagents.
Of Greater Agents and Lesser Agents
How do more sophisticated decision-makers differ from less sophisticated decision-makers in their behaviour and values?
Smarter more sophisticated decisionmakers engage in more and more complex commitments — including meta-commitments not to commit. Consequently, the values and behaviour of these more sophisticated decisionmakers “Greater Agents” are systematically biased compared to less sophisticated decisionmakers “Lesser Agents”.
*******************************
Compared to Lesser Agents, the Greater Agents are more judgemental, (self-)righteous, punish naivité, are more long-term oriented, adaptive, malleable, self-modifying, legibly trustworthy and practice more virtue-signalling, strategic, engage in self-reflection & metacognition, engage in more thinking, less doing, symbolic reasoning, consistent & ‘rational’ in their preferences, they like money & currency more, sacred values less, value engagement in thinking over doing, engaged in more “global” conflicts [including multiverse-wide conflicts throguh acausal trade], less empirical, more rational, more universalistic in their morals, and more cosmopolitan in their esthetics, they are less likely to be threatened, and more willing to martyr themselves, they willing to admit their values’ origins and willing to barter on their values, engage in less frequent but more lethal war, love formal protocols and practice cryptographic magika.
Greater Agents punish Dumb Doves; Greater Agents are Judgemental.
Higher-order cooperators will punish naive lower-order cooperaters for cooperating with bad actors. cf Statistical Physics of Human Cooperation
Greater Agents are more ‘rational’; have more ‘internal cooperation’ and more self-Judgement. They love money and bartering.
They are more coherent & consistent. They more Take Sure Gains and Avoid Sure Losses. See Crystal Healing, Why Not Subagents.
They less adhere to Sacred values; are more willing and able to bargain.
On net, they will use currencies & money more and are more willing to commodify, both internally and externally.
Greater Agents can Super-Cooperate
with twins on the prisoner’s dilemma
others agents with mutual source-code access using Lob’s theorem.
with enough common knowledge of beliefs [upcoming work by N. Stiennon]
Greater Agents practice LessWrong Decision theories
Greater Agents are more threat-resistant, more
but they are still willing to give in to some threats
Greater Agents are upstanding Citizens of the Acausal Society
more universalistic in their moral preferences and more cosmopolitan in their esthetic preferences
Multiversewide Super-rationality by Oesterheld et al.
The Hour I First Believed
Counterintuitively this means Greater Agent instances might fight to the death even if there is (seemingly) nothing to gain in this world. I.e. they might cultivate a Berserker ethos.
Greater Agents one-box on Newcomb’s problem
Greater Agents use some variant of LessWrong decision theories like UDT, FDT, TDT, LDT etc that recommended the ‘correct’ decision in situations where agents are able to engage in sophisticated commitments and simulate other agents with high fidelity.
Greater Agents are Better at Thinking and Meta-Thinking: more malleable, fluid, self-reflective, self-modifying.
Greater Agents are (Meta-)Cognitive Agents
As meta-cognitive agents can outperform less cognitive agents in certain environments Meta-Cognitive Agents will have a bias for thinking (and meta-thinking) over doing.
Trapped cognition Future agents will be more aware of traps in the POMDP of Thought.
Rely more on symbolic, logical, & formal reasoning in place of empirical and intuitive judgements.
Greater Agents are more able to control and harness the environment over long-time horizons; consequently, they have lower time preference and are more long-term oriented.
Greater Agents’ cognition is more ‘adaptive, malleable, fluid’ and they Self-Modify to be more Legibly Trustworthy
Being more legibly trustworthy means they can cooperate more.
Sublety: an upcoming result by Nisan Stiennon says that a class of decisionmakers can (super-rationally) cooperate on the prisoners dilemma if (i) the Lipshitz coefficients of meta-beliefs—i.e. ‘I know that you know that I know that’- satisfy some constaints. Sophisticated decision makers may want to self-modify to constrain their ‘meta-adaptiveness’ (i.e. Lipshitz coefficients).
Greater Agents may or may not get stuck in Commitment races.
Greater Agents Agree and Bargain OR have Hansonian Origin Disputes
Aumann Agreement - rational honest reasoners can never ‘agree to disagree’. They have to Aumann Agree
Real agents aren’t perfect Bayesians. When the true hypothesis is not realizable OR one of the parties has access to privileged indexical information OR the beliefs of the agents are only an approximation of the Bayesian posterior. However, plausibly weaker forms of the Aumann Agreement Theorem may still be applicable.
OR have a value Dispute [Less Frequent more Lethal War]
War is a bargaining failure. Greater Agent employ more sophisticated bargaining solutions. If bargaining fails, wars are more lethal and rapid. They are more lethal for the same reason that forest fires are more rare but lethal when suppressed.
OR they have Origin Disputes.
Origin Disputes may be resolved. For humans this would shake out to more explicitly incorporating evolutionary psychology in one’s moral and explicit reasoning, e.g. admitting and bargaining under the understanding of less-than-perfectly-prosocial evolutionary drives. For synthetically created organisms like AIs instead of evolutionary drives the origin.
OR they may fail to resolve by design. Vanessa Kosoy’s preferred alignment solution is a form of daemonic exorcism that explicitly biases the prior on the prior-origin to rule out exotic problems like Christiano’s Acausal Simulation Attack.
Greater Agents Love and Employ Contracts, Formal Protocols, and Cryptographic Magika
Secure Homes for Digital People
Greater agents may use and write contracts, source-code using exotic cryptographic magic like ZK (Zero-Knowledge proofs); IP (Interactive proof systems); PCP (Probabilitically checkable proofs); Decentralized Trust (‘Blockchain’), Formal Debate (e.g. Double Debate)
Greater Agents are Boltzman/geometrically rational
Sufficiently general Thompson Sampling solves the Grain of Truth problem, enabling enough exploration to approximate Nash equilibria in arbitrary multi-agent environments
Quantal response equilibria see also Direction 15 on Logit equilibria of 2023 Learning-Theoretic Agenda
Or perhaps not. Arithmetic Rationality as a resource? Arithmetic rationality may be better thought of as a resource. Agent-egregores that are more able to bargain across different branches of the multiverse will—i.e. are more able to sacrifice minor loss in one branch for major gain in another brach—collect into more arithmetically rationally superagents.