Mythicists have been told so often (oh how so very often) that they should publish in peer-review journals to be taken seriously. Peer-review, the public has been repeatedly told, is the guarantee of true scholarship. In the recent scholarly outrage over Joseph Atwill’s thesis gaining more public attention than the works of the academy, Larry Hurtado reminded us that “peer-review” and “reputable” go together like carrots and peas and Tom Verenna was once again extolling peer-review as the magic gateway guaranteed to keep intellectuals worthy and honest.
Now I do understand the reasons for peer-review. But what these reminders from among biblical scholars are overlooking is the research that demonstrates that peer-review in the various ways it is practiced today is also a deeply flawed process. Or maybe all those studies demonstrating this have no relevance for theologians.
You see, there is a conflict between what I read on the web by Bible scholars about how effective the peer-review process is in their profession on the one hand, and what I read about the flaws in the peer-review process in my professional capacity (coordinator of a research data management project and of a research publications archiving and access project) in an academic institution on the other.
While I read of the virtues of peer-review for maintaining the pure standards of biblical scholars after hours, during my work time I am reading published research findings that are not so sanguine about peer-review. Why the difference? Could it be that the research is focused mostly in the areas of the sciences? No doubt the nature of that sort of material makes objective analysis easier. Does that mean the demonstrated failings of peer-review could never apply to the field of biblical studies?
Given that scientists are increasingly being exposed to an understanding of the flaws in the peer-review process, are we to assume that biblical scholars are immune from these flaws and that their peer-review mechanisms really are guarantors of quality work?
One article that referenced several studies on the peer-review process is Richard Smith’s “Classical peer review: an empty gun” in Breast Cancer Research 2010, 12 (Suppl 4):S13 doi:10.1186/bcr2742 (a peer-reviewed journal).
If peer review was a drug it would never be allowed onto the market
This is how the article begins. It is a quotation from the deputy editor of a leading medical journal and “intellectual father of the international congresses of peer review that have been held every four years since 1989”, Drummond Rennie.
Later the article makes this claim:
If peer review is to be thought of primarily as a quality assurance method, then sadly we have lots of evidence of its failures. The pretentiously named medical literature is shot through with poor studies.
One would think that in a field like medical studies that peer-review would ensure that only accurate information is published. Certainly we would not think that the peer-review process would let through anything that would cause public harm.
But the facts prove otherwise.
There is much that is published that is downright false. The editors of the ACP Journal Club find that less than 1% of studies in most journals are “both scientifically sound and important for clinicians”. There are also documented instances of bad studies being published that have led to patient heart attacks and measles epidemics.
Note the following and ask if we have the same types of human nature producing and reviewing articles in biblical studies (with my bolding and formatting):
Doug Altman, perhaps the leading expert on statistics in medical journals, sums it up thus: ‘What should we think about researchers who
- use the wrong techniques (either wilfully or in ignorance),
- use the right techniques wrongly,
- misinterpret their results,
- report their results selectively,
- cite the literature selectively,
- and draw unjustified conclusions?’
We should be appalled. Yet numerous studies of the medical literature have shown that all of the above phenomena are common. This is surely a scandal.’
Back to Drummond Rennie:
Drummond Rennie writes in what might be the greatest sentence ever published in a medical journal:
 ‘There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.’ |
Are biblical scholars more professional as a whole than doctors? Are their arguments and publications more rigorous? According to everything I read by biblical scholars themselves I must think they really are. No doubt our souls are worth much more care than our physical bodies.
Richard Smith continues:
We have little or no evidence that peer review ‘works,’ but we have lots of evidence of its downside.
What are the downsides? Continue reading “If Peer-Review Does Not Work for Science Why Does It Work for Biblical Studies?”