2020-08-11

“When everyone is agreed on something, it is probably wrong” — Thompson’s Rule

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

by Neil Godfrey

Another Thompson aphorism: ‘When everyone is agreed on something, it is probably wrong’. In other words, as Thompson has also put it, ‘in our fields, if all are in agreement, it signifies that no one is trying to falsify the theory: an essential step in any scientific argument’. — Doudna 2020

That’s not being perverse. It’s about pausing when “things seem too good to be true” and taking time out to ask if “there has probably been a mistake”. (Gunn, @ 2 mins)

[U]ntil the Romans ultimately removed the right of the Sanhedrin to confer death sentences, a defendant unanimously condemned by the judges would be acquitted [14, Sanhedrin 17a], the Talmud stating ‘If the Sanhedrin unanimously find guilty, he is acquitted. Why? — Because we have learned by tradition that sentence must be postponed till the morrow in hope of finding new points in favour of the defence’.

That practice could be interpreted as the Jewish judges being intuitively aware that suspicions about the process should be raised if the final result appears too perfect . . .

[I]f too many judges agree, the system has failed and should not be considered reliable. (Gunn et al 2016)

Or even more simply,

They intuitively reasoned that when something seems too good to be true, most likely a mistake was made. (Zyga, 2016)

Sanhedrin deciding the death penalty . . . but . . . https://arthive.com/vasilypolenov/works/493225~Guilty_of_death
See Interview 1 and Interview 2 with Thomas L. Thompson. All Vridar blog posts on Thompson’s work are archived here. I expect to begin posting my thoughts on Biblical Narratives, Archaeology & Historicity: Essays in Honour of Thomas L. Thompson fairly soon.

The opening quotation above is from a footnote to a chapter by Gregory Doudna in a newly published volume in honour of Thomas L. Thompson, Biblical Narratives, Archaeology & Historicity: Essays in Honour of Thomas L. Thompson. Doudna’s footnote continues:

I thought of what I have come to call Thompson’s Rule when I encountered this scientific study showing that, as counterintuitive as it sounds, unanimous agreement actually does reduce confidence of correctness in conclusions in a wide variety of disciplines (Gunn et al. 2016).

The paper by Gunn and others is Too good to be true: when overwhelming evidence fails to convince. The argument of the paper (with my bolding in all quotations):

Is it possible for a large sequence of measurements or observations, which support a hypothesis, to counterintuitively decrease our confidence? Can unanimous support be too good to be true? The assumption of independence is often made in good faith; however, rarely is consideration given to whether a systemic failure has occurred. Taking this into account can cause certainty in a hypothesis to decrease as the evidence for it becomes apparently stronger. We perform a probabilistic Bayesian analysis of this effect with examples based on (i) archaeological evidence, (ii) weighing of legal evidence and (iii) cryptographic primality testing. In this paper, we investigate the effects of small error rates in a set of measurements or observations. We find that even with very low systemic failure rates, high confidence is surprisingly difficult to achieve . . . . 

Sometimes as we find more and more agreement we can begin to lose confidence in those results. Gunn begins with a simple example in a presentation he gave in 2016 (link is to youtube video). Here is the key slide:

With a noisy voltmeter attempting to measure a very small voltage (nanovoltage) one would expect some variation in each attempted measurement. Without the variation, we can conclude something is wrong rather than that we have a precise measurement.

Another example:

The recent Volkswagen scandal is a good example. The company fraudulently programmed a computer chip to run the engine in a mode that minimized diesel fuel emissions during emission tests. But in reality, the emissions did not meet standards when the cars were running on the road. The low emissions were too consistent and ‘too good to be true.’ The emissions team that outed Volkswagen initially got suspicious when they found that emissions were almost at the same level whether a car was new or five years old! The consistency betrayed the systemic bias introduced by the nefarious computer chip. (Zyga 2016)

From https://www.cagle.com/arend-van-dam/2015/09/smart-vw-cars

Then there was the Phantom of Heilbronn or the serial killer “Woman Without a Face“. Police spent eight to fifteen years searching for a woman whom DNA connected to 40 crime scenes (murders to burglaries) in France, Germany and Austria. Her DNA was identified at six murder scenes. A three million euro reward was offered. It turned out that the swabs used to collect the DNA from the crime scenes had been inadvertently contaminated at their production point by the same woman.

Consider, also, election results. What do we normally suspect when we hear of a dictator receiving over 90% of the vote?

We have all encountered someone who has argued that “all the evidence” supports their new pet hypothesis to explain, say, Christianity’s origins. I have never been able to persuade them, as far as I know, that reading “all the evidence” with a bias they either cannot see or think is entirely valid.

Ironically, scholars like Bart Ehrman who attempt to deny a historical and even slightly significant “Jesus myth” view among scholars are doing their case a disservice. By insisting that there is and that there has been no valid or reasonable contrary view ever raised, such scholars are undermining confidence in the case for the historicity of Jesus. If they could accept the challenges from serious thinkers over the past near two centuries, and acknowledge the ideological pressure inherent in “biblical studies” for academics to conform within certain parameters of orthodox faith, then they could begin to not look quite so like those politicians who claim 90% of the vote, or like those police chasing a phantom woman serial killer for eight years across Europe, of the dishonest VW executives . . . . Continue reading ““When everyone is agreed on something, it is probably wrong” — Thompson’s Rule”


2013-03-16

Why Bias is Impossible to Avoid and So Hard to Detect

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

by Neil Godfrey

Coincidentally just after posting my last article attempting to demonstrate how bias is inevitable in any research, I started reading an expansion of the same idea in Dennis Nineham’s The Use and Abuse of the Bible.

I will try to minimize the high conceptual jargon and focus on Nineham’s illustrations as he leads the reader step by step into an understanding of the source of our most fundamental and hidden biases. (In all quotations the paragraphing and bolded font is mine. Italics are original.)

To illustrate the way people have rarely been able to distinguish between data or phenomena in the world and their own culturally determined interpretations of it (compare my point in the previous post that some scholars cannot distinguish between raw data and their interpretation of it), Dennis Nineham writes:

Suppose a man lives in a society in which all illness is understood as due to demon-possession. The only questions he will be able to ask about his health — or, mutatis mutandis, about other people’s health — will be: Am I possessed by a demon; and if so, what sort of a demon is it and how can it be expelled?

Because no alternative understanding is available in his community, the symptoms, real of imaginary, which lead him to ask the question will actually be experienced as demonic activity. The pains will feel like a demon biting at his vitals, the rash will be seen as a demon’s claw-marks, and so on. (pp. 4)

Nineham then imagines a slightly more sophisticated situation where the same man lives in a community that permits the existence of an alternative religion that ascribes all sickness to witchcraft.

Even then the understandings of illness available to the man will be limited to these two. He will still not be able, for example, to interpret his symptoms as the end result of a chain of natural causes and effects after the manner of a modern doctor.

What do we mean by “understanding” something?

Nineham then moves to the next point: Continue reading “Why Bias is Impossible to Avoid and So Hard to Detect”