Here’s an idea for a game to train awareness of/resistance to confirmation bias:
The game would consist of three phases, that could then be repeated for however many iterations (levels!) were desired. 1) Presenting and habituating the “theory.” Basically, give a set of rules for making some kind of decision/prediction, and then have the player apply those rules to a series of scenarios that clearly illustrate the supposed predictive (or score-increasing, if you will) properties of the Theory. 2) “In the wild”—Now present a series of scenarios that each either offer evidence that the Theory from phase 1 is useful (+), evidence that the Theory is incorrect(-), or no clear evidence in either direction (null). 3) “Assessment”—Have the player estimate the relative frequencies of (+), (-), and (null) evidence given in phase 2. Player receives an iteration score based on the accuracy of these estimates, and a cumulative score over all iterations completed.
Later iterations (higher levels) could perhaps re-use multiple Theories for the same round, and then in phase 3 ask for evidence estimates for all the Theories at once, possibly even throwing in Theories for which no evidence was presented in the second phase. Higher levels of complexity bring higher stakes (larger increases for accuracy and larger decreases for inaccuracy), so a player who could continue to improve the cumulative score with increases in difficulty would be doing very well indeed.
I’ve spoken of the Theories and Evidence in the purely abstract here, but I’m picturing either color/shape patterns and movements, or plausibly realistic word problem-type scenarios. The former would be preferable since it would not involve importing the player’s biases about situations that might be found in the real world… or actually, come to think of it, it might be interesting and/or useful to make use of realistic-seeming examples precisely for that reason. Huh.
Anyway. The scoring algorithm would reward players who most aggressively sought out (-) evidence for the active Theory or Theories.
I’m not sure I follow how to turn this into a computer game.
Can you give me an in-game example of the “set of rules for making some kind of decision/prediction”? Also the set of scenarios and how to “apply those rules”?
Remember that we have to spell this stuff out for a computer to be able to understand.
This is the simplest sort of example that I was picturing as I wrote the suggestion—it might not be sophisticated enough as described below to be sufficiently challenging.
I also changed my mind a bit about how phase 1 should be structured, so I’ll work that in.
A “scenario” is a box on the screen that is populated by colored shapes that move around like paramecia on a microscope slide, and interact with each other according to the rules for the current round of the game. The scenario ends after a short time period (20-40 seconds) and freezes in its End State. This is what the player will be trying to predict.
Phase 1: Several scenarios are presented in sequence. Each scenario consists of colored shapes interacting with one another—they might bounce off one another and change colors; they might split after a collision or spontaneously; they might eat one another, etc. The interactions show a pattern over the multiple scenarios, such that an observer will eventually start to form predictions about the end state of the system in each scenario. After the pattern has been demonstrated, the player could be asked to code a decision tree for prediction of the end state based on the pattern observed (or this step could actually be skipped, and the Phase 3 predictions just compared to the implicit ruleset used for the pattern without ever making sure the player knows it). Several more scenarios are presented where the player is asked to predict the final state (following the same ruleset as the earlier patterns).
A very simple example of such a ruleset could be as follows:
If there are a circle and a square of the same color, they will collide.
If a red circle collides with a red square, they will each split into two of themselves.
If a blue circle collides with a blue square, the circle will ‘eat’ the square.
If a circle and square of any other color collide, their states will not change after collision.
Phase 2: A given number of scenarios are presented (including the end state). This number is available to the player (ie, the player does not have to keep count of the total numbe (+ evidence). Some explicitly violate these rules (with varying degrees of blatancy—using the ruleset above, one scenario might contain only one pair of shapes that did follow the applicable rule, while another scenario might contain five pairs that misbehaved) (- evidence). Some contain shape/color combos that simply do not contain the right combinations to illustrate the rule (null evidence).
Phase 3: The player is asked to report the relative amounts of (+), (-), and (null) evidence presented in Phase 2.
There is one underlying ruleset per round of the game. Rounds can and should sometimes have rules that contradict rules from previous rounds. The rulesets increase in complexity each time a new round is begun.
Difficulty would increase with complexity of rulesets. Requiring the player to explicitly state the ruleset inferred in Phase 1 would probably make it easier. Introducing interacting symbols that have meaning beyond the bounds of the game (words or pictures) instead of the shapes would likely increase difficulty by requiring the player to ignore prior associations and biases attached to the symbols being used.
We could probably start by coding up a simplified version of this—just to get something done… then add more fo the complex features after that.
For example a good starting point would be for phase 1 predictions to just ask a (randomised) set of multi-choice or simple write-in questions for predictions: eg “how many red squares will there be at the end? in which part of the screen will the blue circle end up?” etc.
I reckon that in the first “level” they could start by estimating a probability, rather than jumping straight into weightings of evidence? We could then introduce evidence weighting as a “level 2″? What do you think? Would that totally change the nature of what it’s teaching too much?
after we’ve got that working, we could then figure out how to get the user to describe the ruleset to the computer in a flexible way. That’s actually a Tough Problem, BTW. It’s basically forming a mini-language… so definitely on the books, but probably not the first iteration. :)
after we’ve got that working, we could then figure out how to get the user to describe the ruleset to the computer in a flexible way. That’s actually a Tough Problem, BTW. It’s basically forming a mini-language… so definitely on the books, but probably not the first iteration. :)
Yeah, I realized that as I was writing the longer example, and also that it wasn’t strictly necessary. Interesting, but not necessary. =)
Your description of phase 1 prediction coding is very close to what I was picturing, and having a randomized set of questions rather than just saying “predict the final state” (in entirety) would give more game repeatability for less code if I understand correctly.
I actually really like the idea of having them just give a probability estimate the first time, or the first few times. I’m betting that will make for an increased effect of confirmation bias in those stages, and that their scores will improve when they’re forced to itemize evidence weights—which illustrates a point about confirmation bias as well as tying into the kind of thought process needed for Bayesian prediction.
(If you were to get as far as trying to code the user-described ruleset bit… I’d suggest finding someone who’s played Dragon Age and ask about the custom tactics options. I think that sort of format would work, as long as the number of total types of game objects and operators stayed relatively small.)
Here’s an idea for a game to train awareness of/resistance to confirmation bias:
The game would consist of three phases, that could then be repeated for however many iterations (levels!) were desired.
1) Presenting and habituating the “theory.” Basically, give a set of rules for making some kind of decision/prediction, and then have the player apply those rules to a series of scenarios that clearly illustrate the supposed predictive (or score-increasing, if you will) properties of the Theory.
2) “In the wild”—Now present a series of scenarios that each either offer evidence that the Theory from phase 1 is useful (+), evidence that the Theory is incorrect(-), or no clear evidence in either direction (null).
3) “Assessment”—Have the player estimate the relative frequencies of (+), (-), and (null) evidence given in phase 2. Player receives an iteration score based on the accuracy of these estimates, and a cumulative score over all iterations completed.
Later iterations (higher levels) could perhaps re-use multiple Theories for the same round, and then in phase 3 ask for evidence estimates for all the Theories at once, possibly even throwing in Theories for which no evidence was presented in the second phase. Higher levels of complexity bring higher stakes (larger increases for accuracy and larger decreases for inaccuracy), so a player who could continue to improve the cumulative score with increases in difficulty would be doing very well indeed.
I’ve spoken of the Theories and Evidence in the purely abstract here, but I’m picturing either color/shape patterns and movements, or plausibly realistic word problem-type scenarios. The former would be preferable since it would not involve importing the player’s biases about situations that might be found in the real world… or actually, come to think of it, it might be interesting and/or useful to make use of realistic-seeming examples precisely for that reason. Huh.
Anyway. The scoring algorithm would reward players who most aggressively sought out (-) evidence for the active Theory or Theories.
That sounds vaguely similar to the game of Eleusis, which I’m surprised wasn’t mentioned yet.
And of course, Zendo.
I’m not sure I follow how to turn this into a computer game. Can you give me an in-game example of the “set of rules for making some kind of decision/prediction”? Also the set of scenarios and how to “apply those rules”?
Remember that we have to spell this stuff out for a computer to be able to understand.
This is the simplest sort of example that I was picturing as I wrote the suggestion—it might not be sophisticated enough as described below to be sufficiently challenging.
I also changed my mind a bit about how phase 1 should be structured, so I’ll work that in.
A “scenario” is a box on the screen that is populated by colored shapes that move around like paramecia on a microscope slide, and interact with each other according to the rules for the current round of the game. The scenario ends after a short time period (20-40 seconds) and freezes in its End State. This is what the player will be trying to predict.
Phase 1: Several scenarios are presented in sequence. Each scenario consists of colored shapes interacting with one another—they might bounce off one another and change colors; they might split after a collision or spontaneously; they might eat one another, etc. The interactions show a pattern over the multiple scenarios, such that an observer will eventually start to form predictions about the end state of the system in each scenario. After the pattern has been demonstrated, the player could be asked to code a decision tree for prediction of the end state based on the pattern observed (or this step could actually be skipped, and the Phase 3 predictions just compared to the implicit ruleset used for the pattern without ever making sure the player knows it). Several more scenarios are presented where the player is asked to predict the final state (following the same ruleset as the earlier patterns).
A very simple example of such a ruleset could be as follows:
If there are a circle and a square of the same color, they will collide.
If a red circle collides with a red square, they will each split into two of themselves.
If a blue circle collides with a blue square, the circle will ‘eat’ the square.
If a circle and square of any other color collide, their states will not change after collision.
Phase 2: A given number of scenarios are presented (including the end state). This number is available to the player (ie, the player does not have to keep count of the total numbe (+ evidence). Some explicitly violate these rules (with varying degrees of blatancy—using the ruleset above, one scenario might contain only one pair of shapes that did follow the applicable rule, while another scenario might contain five pairs that misbehaved) (- evidence). Some contain shape/color combos that simply do not contain the right combinations to illustrate the rule (null evidence).
Phase 3: The player is asked to report the relative amounts of (+), (-), and (null) evidence presented in Phase 2.
There is one underlying ruleset per round of the game. Rounds can and should sometimes have rules that contradict rules from previous rounds. The rulesets increase in complexity each time a new round is begun.
Difficulty would increase with complexity of rulesets. Requiring the player to explicitly state the ruleset inferred in Phase 1 would probably make it easier. Introducing interacting symbols that have meaning beyond the bounds of the game (words or pictures) instead of the shapes would likely increase difficulty by requiring the player to ignore prior associations and biases attached to the symbols being used.
Does that make the idea a bit clearer?
actually yeah—this is a great idea.
We could probably start by coding up a simplified version of this—just to get something done… then add more fo the complex features after that.
For example a good starting point would be for phase 1 predictions to just ask a (randomised) set of multi-choice or simple write-in questions for predictions: eg “how many red squares will there be at the end? in which part of the screen will the blue circle end up?” etc.
I reckon that in the first “level” they could start by estimating a probability, rather than jumping straight into weightings of evidence? We could then introduce evidence weighting as a “level 2″? What do you think? Would that totally change the nature of what it’s teaching too much?
after we’ve got that working, we could then figure out how to get the user to describe the ruleset to the computer in a flexible way. That’s actually a Tough Problem, BTW. It’s basically forming a mini-language… so definitely on the books, but probably not the first iteration. :)
after we’ve got that working, we could then figure out how to get the user to describe the ruleset to the computer in a flexible way. That’s actually a Tough Problem, BTW. It’s basically forming a mini-language… so definitely on the books, but probably not the first iteration. :)
Yeah, I realized that as I was writing the longer example, and also that it wasn’t strictly necessary. Interesting, but not necessary. =)
Your description of phase 1 prediction coding is very close to what I was picturing, and having a randomized set of questions rather than just saying “predict the final state” (in entirety) would give more game repeatability for less code if I understand correctly.
I actually really like the idea of having them just give a probability estimate the first time, or the first few times. I’m betting that will make for an increased effect of confirmation bias in those stages, and that their scores will improve when they’re forced to itemize evidence weights—which illustrates a point about confirmation bias as well as tying into the kind of thought process needed for Bayesian prediction.
(If you were to get as far as trying to code the user-described ruleset bit… I’d suggest finding someone who’s played Dragon Age and ask about the custom tactics options. I think that sort of format would work, as long as the number of total types of game objects and operators stayed relatively small.)