The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I’m not as into predictions as gwern appears to be, but they’re growing on me.
All that and I’m still not very sure what to think of the Knox case. Yes, if our predictions were being scored I’d be getting a non-trivial penalty from my 50% chance of her guilt—that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was “in fact” innocent. (I’m not saying I have much doubt now about her innocence: I’m saying that we won’t ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)
On the other hand, some of the people listed above would be taking a much more serious hit. One thing I’ve learned from my various exercises is that you can’t expect to be right all the time—sometimes, with minimal knowledge of the relevant facts, a 50% prediction is in fact not so bad.
Then again, some of us were also apparently very confident in the answer that “in fact” turned out to be the correct one. Then again, we all get lucky from time to time—that too is the nature of the beast...
My intention is to continue to learn, to continue to get better at predicting, to become better calibrated and more discriminating over time.
(I’m not saying I have much doubt now about her innocence: I’m saying that we won’t ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)
One of the unfortunate things about living when we do is that it seems unlikely there will be any future oracles developed which reveal definitive answers to ancient crimes.
I refer, of course, to DNA evidence, which gave us an astonishing oracle to ask questions about old crimes, revealing a shockingly high lower bound on the justice system’s error rates. If we had been around and recorded predictions about various death row inmates, then the Innocence Project’s &etc. results would’ve been an assessment of our calibration worth writing home about!
One of the unfortunate things about living when we do is that it seems unlikely there will be any future oracles developed which reveal definitive answers to ancient crimes.
I wouldn’t give that an extremely high probability. One of the trends in science is getting surprising amounts of information from tiny amounts of input—DNA is one example, and finding extrasolar planets is another. I don’t have specifics in mind, but I wouldn’t be surprised if another method or two which are at least as powerful are developed.
My reasoning goes along the lines of fingerprints and then DNA are highly precise near unique identifiers, which are also pretty sturdy and accurate. I don’t know of any more biological traces which could significantly improve, much less be orders of magnitude superior to what went before like fingerprints & then DNA were.
There are plenty of future improvements in crime-solving, sure—lifelogging and pervasive surveillance comes to mind as the most obvious improvement. But none of the ones I can think of will be oracles in the sense I mean here of giving us the correct answers for cases we already ‘solved’, none of them will be retrospective. (Lifelogging will be employed as soon as available, witness the Canadian thing with stitching together hundreds of photos online to identify & arrest scores of rioters; I would be surprised if huge archives of recording built up and then only decades later are suddenly made public and cold case units began cracking cases with them, for example.)
Brain scans which can retrieve memories accurately?
I believe that the current understanding is that memory encoding, not just retrieval, is pretty unreliable, so even if you can read exactly what’s in people’s brains, it may not be much help.
And then there’s the problem of locating the right person to interrogate—a task equivalent to and as hard as locating the hypothesis. :) But I probably shouldn’t be too dismissive: brain-scanning is one of the better contenders for the next oracle.
Well, having the right technologies can certainly make locating the hypothesis a lot easier; think how much harder it would have been to locate Guede as a suspect without DNA testing.
If we had a reliable way of determining self perceived truth value, nearly all interrogation could be narrowed down to “Do you know who did it?” “Who did it?” and “Did you do it?”
On second thought, given a device that were capable of doing that with a negligible failure rate, it might be simpler to just replace policing with occasional checkups.
My high school psychology teacher gave my class an interesting demonstration on the usefulness of eyewitnesses.
For one class, we found a notice on the door saying that the class had been relocated to another room. We went to that room, and shortly after our teacher arrived, followed by another teacher. He complained that she had not gotten proper clearance to move her class to that room, and he needed it for an exercise for his own class. They spent a couple minutes arguing, and harsh words were exchanged, after which he left the room.
Almost immediately after he left, our teacher asked us to create profiles of his physical description. Estimates of his height ranged from 5′6 to 6′3, his hair was variously described as being brown, black, or red, and his weight was somewhere from 140 pounds to 230.
The information that the students retrieved from their short term memory, which had not yet been encoded as long term memory, was already profoundly unreliable.
This would probably only work if the person knew activity X was a crime. There are probably ways of becoming convinced that X wasn’t a crime (or vice versa).
This occurred to me, but I suspect that it would still be easier to catch more people this way than with active policing, particularly for the crimes you most care about catching people at, which most people will already know are wrong and probably if not definitely illegal.
I agree with that this method seems far easier to catch the majority of people committing those crimes, since the cultivation process would most likely be non-trivial, so the “casual” criminal wouldn’t have access to it.
There is still the problem of someone else doing the cultivation, so one mastermind could create a militia of people immune to the checkups. (I’m not quite sure if this is actually possible though, although, superficially at least, hypnosis seems to do this sort of thing)
(Thanks for the “cultivating deliberate unawareness” term)
Such a system would be far from most current Western legal systems and hence is of not much interest to me in the context of discovering bounds on error rates in current Western legal systems.
It also occurred to me that (although it’s unlikely) we might discover a neural signature distinguishing spurious memories from accurate ones. Or some less powerful but still useful signature, such as one that distinguishes memories that have been accessed after creation (and so potentially overwritten) from memories that have never been accessed (which are presumably more reliable).
There is weak evidence that the memories you make during the day are reviewed during REM sleep, which would mean every memory is gone over at least once.
The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I’m not as into predictions as gwern appears to be, but they’re growing on me.
All that and I’m still not very sure what to think of the Knox case. Yes, if our predictions were being scored I’d be getting a non-trivial penalty from my 50% chance of her guilt—that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was “in fact” innocent. (I’m not saying I have much doubt now about her innocence: I’m saying that we won’t ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)
On the other hand, some of the people listed above would be taking a much more serious hit. One thing I’ve learned from my various exercises is that you can’t expect to be right all the time—sometimes, with minimal knowledge of the relevant facts, a 50% prediction is in fact not so bad.
Then again, some of us were also apparently very confident in the answer that “in fact” turned out to be the correct one. Then again, we all get lucky from time to time—that too is the nature of the beast...
My intention is to continue to learn, to continue to get better at predicting, to become better calibrated and more discriminating over time.
One of the unfortunate things about living when we do is that it seems unlikely there will be any future oracles developed which reveal definitive answers to ancient crimes.
I refer, of course, to DNA evidence, which gave us an astonishing oracle to ask questions about old crimes, revealing a shockingly high lower bound on the justice system’s error rates. If we had been around and recorded predictions about various death row inmates, then the Innocence Project’s &etc. results would’ve been an assessment of our calibration worth writing home about!
I wouldn’t give that an extremely high probability. One of the trends in science is getting surprising amounts of information from tiny amounts of input—DNA is one example, and finding extrasolar planets is another. I don’t have specifics in mind, but I wouldn’t be surprised if another method or two which are at least as powerful are developed.
My reasoning goes along the lines of fingerprints and then DNA are highly precise near unique identifiers, which are also pretty sturdy and accurate. I don’t know of any more biological traces which could significantly improve, much less be orders of magnitude superior to what went before like fingerprints & then DNA were.
There are plenty of future improvements in crime-solving, sure—lifelogging and pervasive surveillance comes to mind as the most obvious improvement. But none of the ones I can think of will be oracles in the sense I mean here of giving us the correct answers for cases we already ‘solved’, none of them will be retrospective. (Lifelogging will be employed as soon as available, witness the Canadian thing with stitching together hundreds of photos online to identify & arrest scores of rioters; I would be surprised if huge archives of recording built up and then only decades later are suddenly made public and cold case units began cracking cases with them, for example.)
Brain scans which can retrieve memories accurately? Admittedly, this would be limited to crimes with living witnesses.
The thing is, I think science leads to weird, surprising discoveries. I’m not going to predict what’s impossible a century from now.
I believe that the current understanding is that memory encoding, not just retrieval, is pretty unreliable, so even if you can read exactly what’s in people’s brains, it may not be much help.
And then there’s the problem of locating the right person to interrogate—a task equivalent to and as hard as locating the hypothesis. :) But I probably shouldn’t be too dismissive: brain-scanning is one of the better contenders for the next oracle.
Well, having the right technologies can certainly make locating the hypothesis a lot easier; think how much harder it would have been to locate Guede as a suspect without DNA testing.
If we had a reliable way of determining self perceived truth value, nearly all interrogation could be narrowed down to “Do you know who did it?” “Who did it?” and “Did you do it?”
On second thought, given a device that were capable of doing that with a negligible failure rate, it might be simpler to just replace policing with occasional checkups.
“So, committed any crimes this month?”
Self-perceived truth value sounds like a subtle problem.
I’d settle for memory testing for eye-witnesses, though memory testing under stress is probably too much to ask.
My high school psychology teacher gave my class an interesting demonstration on the usefulness of eyewitnesses.
For one class, we found a notice on the door saying that the class had been relocated to another room. We went to that room, and shortly after our teacher arrived, followed by another teacher. He complained that she had not gotten proper clearance to move her class to that room, and he needed it for an exercise for his own class. They spent a couple minutes arguing, and harsh words were exchanged, after which he left the room.
Almost immediately after he left, our teacher asked us to create profiles of his physical description. Estimates of his height ranged from 5′6 to 6′3, his hair was variously described as being brown, black, or red, and his weight was somewhere from 140 pounds to 230.
The information that the students retrieved from their short term memory, which had not yet been encoded as long term memory, was already profoundly unreliable.
This would probably only work if the person knew activity X was a crime. There are probably ways of becoming convinced that X wasn’t a crime (or vice versa).
This occurred to me, but I suspect that it would still be easier to catch more people this way than with active policing, particularly for the crimes you most care about catching people at, which most people will already know are wrong and probably if not definitely illegal.
And you could always ask them, “Are you cultivating deliberate unawareness?”
The whole point of deliberate unawareness is that it’s unaware—the deliberate part gets hidden in mental fog.
I agree with that this method seems far easier to catch the majority of people committing those crimes, since the cultivation process would most likely be non-trivial, so the “casual” criminal wouldn’t have access to it.
There is still the problem of someone else doing the cultivation, so one mastermind could create a militia of people immune to the checkups. (I’m not quite sure if this is actually possible though, although, superficially at least, hypnosis seems to do this sort of thing)
(Thanks for the “cultivating deliberate unawareness” term)
Such a system would be far from most current Western legal systems and hence is of not much interest to me in the context of discovering bounds on error rates in current Western legal systems.
I thought the same thing.
It also occurred to me that (although it’s unlikely) we might discover a neural signature distinguishing spurious memories from accurate ones. Or some less powerful but still useful signature, such as one that distinguishes memories that have been accessed after creation (and so potentially overwritten) from memories that have never been accessed (which are presumably more reliable).
There is weak evidence that the memories you make during the day are reviewed during REM sleep, which would mean every memory is gone over at least once.
Even just reliable brain-scan based lie detectors...