Evaluating Evidence Reconstructions of Mock Crimes -Submission 2

Epistemic Status: Exploratory

Introduction

I want to train people to decide the truth about events, through practices reconstructing events about which the truth is independently known

I am asking the help of people who are interested in applying reason to understand how reason and the understanding of it should shape how training for applying reason to solve a class of practical problems should be evaluated

2. At present:

-no one is specifically trained to bring disputes about the facts of events to a final conclusion

-we have no way to tell how accurate factfinders or factfinding methods are

-dispute resolution is shuffled to people not specifically trained or designated for it

3. Alternatives considered:

-common sense -anyone can do it

-learning through experience -but what feedback?

-academic professional education -how could effect of this be known

4. Benefits of using practices to train:

we could determine proxies for base rates of factfinding accuracy

we could tell whether and which training improved factfinding ability

we could select more accurate factfinders to do the job

5. Question:

Do you have any ideas for how this should be done? What should a student’s results be judged on, compared to? What features should they be marked on?

What in a prior recording of the event should be considered salient

6. What I have typed out so far about practices:

Factfinding Practices

A. general idea

i. tests/exercise, where mock crime has been committed by one of n people, or by any of whole class or more, and whether student (first researcher?) can find out which

ii. people coming to them with true and false problems, prizes for those who can successfully fake (true problems referred from an actual resolution service? That circular?)

iii. also use natural occurrences in classes, on campus -where recording? Provision for that? [or without recording just comparing the findings of different fact finders]

B. Specifics

i. to be dispersed through whole three year’s training independently of how subjects otherwise organized

ii. to be sometimes done with other classes than those for being justice agent, to get more of general population distribution of traits, responses ….

iii. Collect students’ estimates of how long such a factfinding should take (estimates for different subjects, contexts?) and try allotting time in (different) amounts estimated, whether more wanted, how often task thought done in less -compare results from different lengths

C. evaluation:

i. results to be compared to at least 3 recordings from different vantage points -repetition of vantage points could lead to strategizing for them -how that could happen otherwise?

ii. purposes to evaluate factfinding performance as serving or not: to find out a/ who did something b/ was something done in self-defense or unjustifiably c/ was something done voluntarily

iii. at least some to be done by multiple raters, methods and (dis) agreement among those tracked, analysed

iv. The {measures _} should be studied as a whole to determine their distribution across person-practices and persons, then what that distribution implies and how it compares to ones that would be expected on various assumptions, including randomness, investigated, and the probability of randomly getting a distribution at least as improbable as the one observed calculated.

v. evaluations should be kept and correlated with subsequent black marks ?and other performance measures (studied as in iv/) of students who become justice agents -in detail with each agents distribution of times from invocation to resolution of a matter

vi .time consumed by and time from beginning to end of each factfinding to be tracked, recorded, analysed, perhaps marked on

vii. tests of ability to notice, realize something has been done (probably not omitted)

D. Further

i. costing how much -initial v. ongoing, experiment, will be setting u

7. Background:

As a partial and temporary remedy for the ongoing famine of public adjudication in Canada and other countries, in 2018 I proposed the creation of the justice agent, an official intermediate between the present police officers, mediators, and lowest level trial court judges. I sent this proposal to the Canadian and Ontario Ministers of Justice and other possibly interested parties.

I hope to publish the proposal on a website I will start, perhaps in 2025, and in particular to propose that it be implemented as a ten-year1 pilot project.

To support this, I thought I would develop two aspects of the proposal in further detail. The aspect I have actually worked on how is how the agents would be trained/educated for their work.

I now think that factfinding practices themselves could be done as an experimental project, testing among other things whether those who claim to have expertise from experience or academic education -or everyone- in fact gets the relevant facts right almost always.

1 For 10 years of operation, with 3 years of training the first agents to precede that and whether to continue the training in the last 3 years of operation to be decided then on the basis of what it was then thought to do going forward.