I’m not going to talk about option 2 because it stops people from being perfect reasoners. (If there’s a subset of option 2 that still lets people be perfect reasoners, I’d love to hear it—that might be the most interesting part of the puzzle). That leaves option 1.
Here’s a simple model of option 1. Nature shuffles a deck of cards randomly, then a sorcerer (if one exists) has a chance to rearrange the cards somehow, then the deck is shown to an observer, who uses it as Bayesian evidence for or against the sorcerer’s existence. We will adopt the usual “Nash equilibrium” assumption that the observer knows the sorcerer’s strategy in advance. This seems like a fair idealization of “moving things around in the world”. What would the different types of sorcerers do?
Note that if both Bright and Dark might exist, the game becomes unpleasant to analyze, because Dark can try to convince the observer that Bright exists, which would mean Dark doesn’t exist. To simplify the game, we will let the observer know which type of sorcerer they might be playing against, so they only need to determine if the sorcerer exists.
A (non-unique) best strategy for Bright is to rearrange the cards in perfect order, so the observer can confidently say “either Bright exists or I just saw a very improbable coincidence”. A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer’s prior. Invisible has the same set of best strategies as Dark. I won’t spell out the proofs here, anyone sufficiently interested should be able to work them out.
To summarize: if sorcerers can only move things around in the world and cannot influence people’s minds directly, then Bright does as much as possible, Invisible and Dark do as little as possible, and the observer only looks at things in the world and doesn’t do anything like “updating on the strength of their own beliefs”. The latter is only possible if sorcerers can directly influence minds, which stops people from being perfect reasoners and is probably harder to model and analyze.
Overall it seems like your post can generate several interesting math problems, depending on how you look at it. Good work!
Every person in faeri has a Phsycomagical Intuition. This has a 50% probability of always detecting if there are any sorcerers within a light minute or so, and a 50% probability of giving random results (that don’t change or otherwise can be distinguished from having a functioning one). A sorcerer can expend some effort to set the non-functioning ones to whatever it wants.
The sorcerer cannot affect existing minds, but can alter any faerians priors before birth, in a probabilistic manner within normal human variation.
Hmm, the first case seems reducible to “moving things around in the world”, and the second sounds like it might be solvable by Robin Hanson’s pre-rationality.
How about, if Bob has a sort of “sorcerous experience” which is kind of like an epiphany. I don’t want to go off to Zombie-land with this, but let’s say it could be caused by his brain doing its mysterious thing, or by a sorcerer. Does that still count as “moving things around in the world”?
Well, it seems possible to set up an equivalent game (with the same probabilities etc) where the sorcerer is affecting a card deck that’s shown to you.
Maybe I should have drawn the distinction differently. If the sorcerer can only affect your experiences, that’s basically the same as affecting a card deck. But if the sorcerer can affect the way you process these experiences, e.g. force you to not do a Bayesian update where you normally would, or reach into your mind and make you think you had a different prior all along, that’s different because it makes you an imperfect reasoner. We know how to answer questions like “what should a perfect reasoner do?” but we don’t know much about “what should such-and-such imperfect reasoner do?”
I see what you mean now, I think. I don’t have a good model of dealing with a situation where someone can influence the actual updating process either. I was always thinking of a setup where the sorcerer affects something other than this.
By the way, I remember reading a book which had a game-theoretical analysis of games where one side had god-like powers (omniscience, etc), but I don’t remember what it was called. Does anyone reading this by any chance know which book I mean?
A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer’s prior
If I were a Dark, I would try to rearrange the cards so they look random to an unsophisticated observer. No long runs of same color, no obvious patterns in numbers (people are bad random number generators, they think that random string is string without any patterns, not string without big patterns, 17 is the most random number, blah blah blah).
(It’s possible that the variation of it can be a good strategy even against more sophisticated agents, because if by a pure chance string of cards has low Kolmogorov complexity, agent is going to take this as evidence for Bright, and I don’t want him to believe in Bright)
I think I have a proof that the only Nash equilibrium strategies for Dark playing against a perfect reasoner are those that lead to a uniform distribution over observed decks. K-complexity doesn’t seem to come into it. What Dark should do against an imperfect reasoner is a different question, which we can’t solve because we don’t have a good theory of imperfect reasoning.
Sorcerers could influence people’s minds to just change their goals/utility functions without stopping them from being perfect reasoners. This is why people worry so much about AI friendliness, because assuming infinite computational capacity there is still (we assume) no privileged utility function.
Looks like my argument leads to a mildly interesting result. Let’s say a Bayesian is playing a game against a Tamperer. The Bayesian is receiving evidence about something, and tries to get more accurate beliefs about that thing according to some proper scoring rule. The Tamperer sits in the middle, tampering with the evidence received by the Bayesian. Then any Nash equilibrium will give the Bayesian an expected score that’s at least as high as they would’ve got by just using their prior and ignoring the evidence. (The expected score is calculated according to the Bayesian’s prior.) In other words, you cannot deliberately lead a Bayesian away from the truth.
The proof is kinda trivial: the Bayesian can guarantee a certain score by just using the prior, regardless of what the Tamperer does. Therefore in any Nash equilibrium the Bayesian will get at least that much, and might get more if the Tamperer’s opportunities for tampering are somehow limited.
That rather depends on whether the Bayesian (usually known as Bob) knows there is a Tamperer (usually known as Mallory) messing around with his evidence.
If the Bayesian does know, he just distrusts all evidence and doesn’t move off his prior. But if he does not know, then the Tamperer just pwns him.
I think your objection is kinda covered by the use of the term “Nash equilibrium” in my comment. And even if the universe decides to create a Tamperer with some probability and leave the evidence untouched otherwise, the result should still hold. The term for that kind of situation is “Bayes-Nash equilibrium”, I think.
Bob is playing a zero-sum game against Mallory. All Bob’s information is filtered/changed/provided by Mallory and Bob knows it. In this situation Bob cannot trust any of this information and so never changes his response or belief.
The result also applies if Mallory has limited opportunities to change Bob’s information, e.g. a 10% chance of successfully changing it. Or you could have any other complicated setup. In such cases Bob’s best strategy involves some updating, and the result says that such updating cannot lower Bob’s score on average. (If you’re wondering why Bob’s strategy in a Nash equilibrium must look like Bayesian updating at all, the reason is given by the definition of a proper scoring rule.) In other words, it’s still trivial, but not quite as trivial as you say. Also note that if Mallory’s options are limited, her best strategy might become pretty complicated.
If there’s a subset of option 2 that still lets people be perfect reasoners, I’d love to hear it—that might be the most interesting part of the puzzle
Make them forget about some piece of evidence they already updated on! Let’s say that evidence A moves B in some direction, and that P(B|A) has already been computed. If you forget A, by encountering A again you would get P(B|A,A) <> P(B|A), while still executing the perfect reasoner’s algorithm.
People can counteract that trick and other similar tricks by constantly regenerating their beliefs from their original prior and remembered evidence. Can you make a more watertight model?
I think we can combine your [cousin_it’s] suggestion with MrMind’s for an Option 2 scenario.
Suppose Bob finds that he has a stored belief in Bright with an apparent memory of having based it on evidence A, but no memory of what evidence A was. That does constitute some small evidence in favor of Bright existing.
But if Bob then goes out in search of evidence about whether Bright exists, and finds some evidence A in favor, he is unable to know whether it’s the same evidence as before that he had forgotten, or if it’s different evidence. Another way of saying that is that Bob can’t tell whether or not A and A are independent. I suppose the ideal reasoner’s response would be to assign a probability density distribution over a range from full independence to full dependence and proceed with any belief updates taking that distribution into account.
The distribution should be formed by consideration of how Bob got the evidence. If Bob found his new evidence A in some easily repeatable way, like hearing it from Bright apologists, then Bob would probably think dependence on A is much more likely than independence, and so he would take into account mostly just A and not A. But if Bob got A by some means that he probably wouldn’t have had access to in the past, like an experiment requiring brand new technology to perform, then he would probably think independence was more likely, and so he would take into account A and A mostly separately.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior (at least from the subjective view of an external observer) by ridding them (possibly temporarily) of key facts.
And then there is the question of whether they may notice this as some inferences are more likely to be detected when you already have some other facts.
I’d guess that you’d quickly notice if you should suddenly have forgotten that you were repeatedly told that Bright exists.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior
Surely I can construct such a model. But whether this is generally the case depends too much on the details of the implementation to give a complete answers.
whether they may notice this as some inferences are more likely to be detected when you already have some other facts.
… especially logical inferences: logical deductions of true facts are true, even if you don’t know/remember them. But then again, that depends too much on the implementation of the agent to have a general answer, in this case also its computational power would matter.
That’s a very interesting analysis. I think you are taking the point of view that sorcerers are rational, or that they are optimizing solely for proving or disproving their existence. That wasn’t my assumption. Sorcerers are mysterious, so people can’t expect their cooperation in an experiment designed for this purpose. Even under your assumption you can never distinguish between Bright and Dark existing: they could behave identically, to convince you that Bright exists. Dark would sort the deck whenever you query for Bright, for instance.
The way I was thinking about it is that you have other beliefs about sorcerers and your evidence for their existence is primarily established based on other grounds (e.g. see my comment about kittens in another thread). Then Bob and Daisy take into account the fact that Bright and Dark have these additional peculiar preferences for people’s belief in them.
The assumption of rationality is usually used to get a tractable game. That said, the assumption is not as restrictive as you seem to say. A rational sorcerer isn’t obliged to cooperate with you, and can have other goals as well. For example, in my game we could give Dark a strong desire to move the ace of spades to the top of the deck, and that desire could have a certain weight compared to the desire to stay hidden. In the resulting game, Daisy would still use only the information from the deck, and wouldn’t need to do Bayesian updates based on her own state of mind. Does that answer your question?
A sorcerer has two ways to manipulate people:
1) Move things around in the world.
2) Directly influence people’s minds.
I’m not going to talk about option 2 because it stops people from being perfect reasoners. (If there’s a subset of option 2 that still lets people be perfect reasoners, I’d love to hear it—that might be the most interesting part of the puzzle). That leaves option 1.
Here’s a simple model of option 1. Nature shuffles a deck of cards randomly, then a sorcerer (if one exists) has a chance to rearrange the cards somehow, then the deck is shown to an observer, who uses it as Bayesian evidence for or against the sorcerer’s existence. We will adopt the usual “Nash equilibrium” assumption that the observer knows the sorcerer’s strategy in advance. This seems like a fair idealization of “moving things around in the world”. What would the different types of sorcerers do?
Note that if both Bright and Dark might exist, the game becomes unpleasant to analyze, because Dark can try to convince the observer that Bright exists, which would mean Dark doesn’t exist. To simplify the game, we will let the observer know which type of sorcerer they might be playing against, so they only need to determine if the sorcerer exists.
A (non-unique) best strategy for Bright is to rearrange the cards in perfect order, so the observer can confidently say “either Bright exists or I just saw a very improbable coincidence”. A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer’s prior. Invisible has the same set of best strategies as Dark. I won’t spell out the proofs here, anyone sufficiently interested should be able to work them out.
To summarize: if sorcerers can only move things around in the world and cannot influence people’s minds directly, then Bright does as much as possible, Invisible and Dark do as little as possible, and the observer only looks at things in the world and doesn’t do anything like “updating on the strength of their own beliefs”. The latter is only possible if sorcerers can directly influence minds, which stops people from being perfect reasoners and is probably harder to model and analyze.
Overall it seems like your post can generate several interesting math problems, depending on how you look at it. Good work!
some models of 2:
Every person in faeri has a Phsycomagical Intuition. This has a 50% probability of always detecting if there are any sorcerers within a light minute or so, and a 50% probability of giving random results (that don’t change or otherwise can be distinguished from having a functioning one). A sorcerer can expend some effort to set the non-functioning ones to whatever it wants.
The sorcerer cannot affect existing minds, but can alter any faerians priors before birth, in a probabilistic manner within normal human variation.
Hmm, the first case seems reducible to “moving things around in the world”, and the second sounds like it might be solvable by Robin Hanson’s pre-rationality.
How about, if Bob has a sort of “sorcerous experience” which is kind of like an epiphany. I don’t want to go off to Zombie-land with this, but let’s say it could be caused by his brain doing its mysterious thing, or by a sorcerer. Does that still count as “moving things around in the world”?
Well, it seems possible to set up an equivalent game (with the same probabilities etc) where the sorcerer is affecting a card deck that’s shown to you.
Maybe I should have drawn the distinction differently. If the sorcerer can only affect your experiences, that’s basically the same as affecting a card deck. But if the sorcerer can affect the way you process these experiences, e.g. force you to not do a Bayesian update where you normally would, or reach into your mind and make you think you had a different prior all along, that’s different because it makes you an imperfect reasoner. We know how to answer questions like “what should a perfect reasoner do?” but we don’t know much about “what should such-and-such imperfect reasoner do?”
I see what you mean now, I think. I don’t have a good model of dealing with a situation where someone can influence the actual updating process either. I was always thinking of a setup where the sorcerer affects something other than this.
By the way, I remember reading a book which had a game-theoretical analysis of games where one side had god-like powers (omniscience, etc), but I don’t remember what it was called. Does anyone reading this by any chance know which book I mean?
You might be thinking of Superior Beings by Steven Brams.
(My favourite result of this kind is that if you play Chicken with God, then God loses.)
If I were a Dark, I would try to rearrange the cards so they look random to an unsophisticated observer. No long runs of same color, no obvious patterns in numbers (people are bad random number generators, they think that random string is string without any patterns, not string without big patterns, 17 is the most random number, blah blah blah).
(It’s possible that the variation of it can be a good strategy even against more sophisticated agents, because if by a pure chance string of cards has low Kolmogorov complexity, agent is going to take this as evidence for Bright, and I don’t want him to believe in Bright)
I think I have a proof that the only Nash equilibrium strategies for Dark playing against a perfect reasoner are those that lead to a uniform distribution over observed decks. K-complexity doesn’t seem to come into it. What Dark should do against an imperfect reasoner is a different question, which we can’t solve because we don’t have a good theory of imperfect reasoning.
Sorcerers could influence people’s minds to just change their goals/utility functions without stopping them from being perfect reasoners. This is why people worry so much about AI friendliness, because assuming infinite computational capacity there is still (we assume) no privileged utility function.
Looks like my argument leads to a mildly interesting result. Let’s say a Bayesian is playing a game against a Tamperer. The Bayesian is receiving evidence about something, and tries to get more accurate beliefs about that thing according to some proper scoring rule. The Tamperer sits in the middle, tampering with the evidence received by the Bayesian. Then any Nash equilibrium will give the Bayesian an expected score that’s at least as high as they would’ve got by just using their prior and ignoring the evidence. (The expected score is calculated according to the Bayesian’s prior.) In other words, you cannot deliberately lead a Bayesian away from the truth.
The proof is kinda trivial: the Bayesian can guarantee a certain score by just using the prior, regardless of what the Tamperer does. Therefore in any Nash equilibrium the Bayesian will get at least that much, and might get more if the Tamperer’s opportunities for tampering are somehow limited.
That rather depends on whether the Bayesian (usually known as Bob) knows there is a Tamperer (usually known as Mallory) messing around with his evidence.
If the Bayesian does know, he just distrusts all evidence and doesn’t move off his prior. But if he does not know, then the Tamperer just pwns him.
I think your objection is kinda covered by the use of the term “Nash equilibrium” in my comment. And even if the universe decides to create a Tamperer with some probability and leave the evidence untouched otherwise, the result should still hold. The term for that kind of situation is “Bayes-Nash equilibrium”, I think.
In this case what’s special about Bayesians here?
Bob is playing a zero-sum game against Mallory. All Bob’s information is filtered/changed/provided by Mallory and Bob knows it. In this situation Bob cannot trust any of this information and so never changes his response or belief.
I don’t see any reason to invoke St.Bayes.
The result also applies if Mallory has limited opportunities to change Bob’s information, e.g. a 10% chance of successfully changing it. Or you could have any other complicated setup. In such cases Bob’s best strategy involves some updating, and the result says that such updating cannot lower Bob’s score on average. (If you’re wondering why Bob’s strategy in a Nash equilibrium must look like Bayesian updating at all, the reason is given by the definition of a proper scoring rule.) In other words, it’s still trivial, but not quite as trivial as you say. Also note that if Mallory’s options are limited, her best strategy might become pretty complicated.
Make them forget about some piece of evidence they already updated on!
Let’s say that evidence A moves B in some direction, and that P(B|A) has already been computed. If you forget A, by encountering A again you would get P(B|A,A) <> P(B|A), while still executing the perfect reasoner’s algorithm.
People can counteract that trick and other similar tricks by constantly regenerating their beliefs from their original prior and remembered evidence. Can you make a more watertight model?
I think we can combine your [cousin_it’s] suggestion with MrMind’s for an Option 2 scenario.
Suppose Bob finds that he has a stored belief in Bright with an apparent memory of having based it on evidence A, but no memory of what evidence A was. That does constitute some small evidence in favor of Bright existing.
But if Bob then goes out in search of evidence about whether Bright exists, and finds some evidence A in favor, he is unable to know whether it’s the same evidence as before that he had forgotten, or if it’s different evidence. Another way of saying that is that Bob can’t tell whether or not A and A are independent. I suppose the ideal reasoner’s response would be to assign a probability density distribution over a range from full independence to full dependence and proceed with any belief updates taking that distribution into account.
The distribution should be formed by consideration of how Bob got the evidence. If Bob found his new evidence A in some easily repeatable way, like hearing it from Bright apologists, then Bob would probably think dependence on A is much more likely than independence, and so he would take into account mostly just A and not A. But if Bob got A by some means that he probably wouldn’t have had access to in the past, like an experiment requiring brand new technology to perform, then he would probably think independence was more likely, and so he would take into account A and A mostly separately.
But I wonder whether you could manipulate them this way arbitrarily far from rational behavior (at least from the subjective view of an external observer) by ridding them (possibly temporarily) of key facts.
And then there is the question of whether they may notice this as some inferences are more likely to be detected when you already have some other facts. I’d guess that you’d quickly notice if you should suddenly have forgotten that you were repeatedly told that Bright exists.
Surely I can construct such a model. But whether this is generally the case depends too much on the details of the implementation to give a complete answers.
… especially logical inferences: logical deductions of true facts are true, even if you don’t know/remember them. But then again, that depends too much on the implementation of the agent to have a general answer, in this case also its computational power would matter.
That’s a very interesting analysis. I think you are taking the point of view that sorcerers are rational, or that they are optimizing solely for proving or disproving their existence. That wasn’t my assumption. Sorcerers are mysterious, so people can’t expect their cooperation in an experiment designed for this purpose. Even under your assumption you can never distinguish between Bright and Dark existing: they could behave identically, to convince you that Bright exists. Dark would sort the deck whenever you query for Bright, for instance.
The way I was thinking about it is that you have other beliefs about sorcerers and your evidence for their existence is primarily established based on other grounds (e.g. see my comment about kittens in another thread). Then Bob and Daisy take into account the fact that Bright and Dark have these additional peculiar preferences for people’s belief in them.
The assumption of rationality is usually used to get a tractable game. That said, the assumption is not as restrictive as you seem to say. A rational sorcerer isn’t obliged to cooperate with you, and can have other goals as well. For example, in my game we could give Dark a strong desire to move the ace of spades to the top of the deck, and that desire could have a certain weight compared to the desire to stay hidden. In the resulting game, Daisy would still use only the information from the deck, and wouldn’t need to do Bayesian updates based on her own state of mind. Does that answer your question?