Jim West, a biblical studies pastor whose name sometimes appears in publications alongside the likes of Thomas L. Thompson and Niels Peter Lemche, and who also runs a blog that I follow for its occasional titbits of news on recent publications, did not earn his academic qualifications at an accredited university, if I understand correctly, and periodically rails against accreditation itself. In a recent post he also attacks “scientific journals” for accepting a bogus paper for publication and uses this as an opportunity to rail against the value of peer review.
Don’t be misled, however. The article to which he refers is discussing makes clear are “predatory publishers” — that is, scam publishers who are not serious academic publishers at all but the same sort of scams as you get in your email from Nigerian bankers offering you millions from lost accounts, etc.
A list of these predatory journal titles is kept at Beall’s List of Predatory Journals and Publishers. They prey on the new academics, still wet behind their ears and eager to get published. — Or they prey on old academics like Jim West who isolates himself from standards of accreditation etc and is so keeps himself out of the loop, it seems.
The point is, don’t let anyone think that a bogus paper being published by a “scientific journal” that is actually a “predatory” title is kind of evidence, for or against, peer review. These publishers in all likelihood don’t have peer review at all — or lie if they say they do — and probably don’t even bother to have anyone read papers that they publish.
Sian Harris has made some interesting comments about current trends in scholarly publishing, including observations of what is happening to peer-review.
On peer-review (my bolding) one of the developments noted is:
Peer review is another interesting trend to watch. Different journals take different approaches to this. One trend is from blind to open peer review, where authors and reviewers know each other’s identity. Another topic for discussion is whether the lion’s share of peer review should go on pre-publication or post-publication. The journal PLOS One has an interesting approach to this, of deliberately only assessing papers for things like originality, accuracy and ethics but not making a judgement on how interesting the research is before publication. The discussion of the value of the research goes on afterwards.
(Not that any of this is necessary for the Humanities, according to Larry Hurtado. Everything is just fine there. It’s only where those scientists have the ability to fabricate data that we find any problems, according to his ostrich perspective.)
We have little or no evidence that peer review ‘works,’ but we have lots of evidence of its downside.
The Gatekeeper at the entrance to the Emerald City (Photo credit: Wikipedia)
Mythicists have been told so often (oh how so very often) that they should publish in peer-review journals to be taken seriously. Peer-review, the public has been repeatedly told, is the guarantee of true scholarship. In the recent scholarly outrage over Joseph Atwill’s thesis gaining more public attention than the works of the academy, Larry Hurtado reminded us that “peer-review” and “reputable” go together like carrots and peas and Tom Verenna was once again extolling peer-review as the magic gateway guaranteed to keep intellectuals worthy and honest.
Now I do understand the reasons for peer-review. But what these reminders from among biblical scholars are overlooking is the research that demonstrates that peer-review in the various ways it is practiced today is also a deeply flawed process. Or maybe all those studies demonstrating this have no relevance for theologians.
You see, there is a conflict between what I read on the web by Bible scholars about how effective the peer-review process is in their profession on the one hand, and what I read about the flaws in the peer-review process in my professional capacity (coordinator of a research data management project and of a research publications archiving and access project) in an academic institution on the other.
While I read of the virtues of peer-review for maintaining the pure standards of biblical scholars after hours, during my work time I am reading published research findings that are not so sanguine about peer-review. Why the difference? Could it be that the research is focused mostly in the areas of the sciences? No doubt the nature of that sort of material makes objective analysis easier. Does that mean the demonstrated failings of peer-review could never apply to the field of biblical studies?
Given that scientists are increasingly being exposed to an understanding of the flaws in the peer-review process, are we to assume that biblical scholars are immune from these flaws and that their peer-review mechanisms really are guarantors of quality work?
One article that referenced several studies on the peer-review process is Richard Smith’s “Classical peer review: an empty gun” in Breast Cancer Research 2010, 12 (Suppl 4):S13 doi:10.1186/bcr2742 (a peer-reviewed journal).
If peer review was a drug it would never be allowed onto the market
This is how the article begins. It is a quotation from the deputy editor of a leading medical journal and “intellectual father of the international congresses of peer review that have been held every four years since 1989”, Drummond Rennie.
Later the article makes this claim:
If peer review is to be thought of primarily as a quality assurance method, then sadly we have lots of evidence of its failures. The pretentiously named medical literature is shot through with poor studies.
One would think that in a field like medical studies that peer-review would ensure that only accurate information is published. Certainly we would not think that the peer-review process would let through anything that would cause public harm.
But the facts prove otherwise.
There is much that is published that is downright false. The editors of the ACP Journal Club find that less than 1% of studies in most journals are “both scientifically sound and important for clinicians”. There are also documented instances of bad studies being published that have led to patient heart attacks and measles epidemics.
Note the following and ask if we have the same types of human nature producing and reviewing articles in biblical studies (with my bolding and formatting):
Doug Altman, perhaps the leading expert on statistics in medical journals, sums it up thus: ‘What should we think about researchers who
use the wrong techniques (either wilfully or in ignorance),
use the right techniques wrongly,
misinterpret their results,
report their results selectively,
cite the literature selectively,
and draw unjustified conclusions?’
We should be appalled. Yet numerous studies of the medical literature have shown that all of the above phenomena are common. This is surely a scandal.’
Back to Drummond Rennie:
Drummond Rennie writes in what might be the greatest sentence ever published in a medical journal:
D. Rennie
‘There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.’
Are biblical scholars more professional as a whole than doctors? Are their arguments and publications more rigorous? According to everything I read by biblical scholars themselves I must think they really are. No doubt our souls are worth much more care than our physical bodies.
Richard Smith continues:
We have little or no evidence that peer review ‘works,’ but we have lots of evidence of its downside.
Part of my job as a coordinator of the management of research data at an Australian university is to be familiar with the Australian Code for the Responsible Conduct of Research. The section in this document on Peer Review (chapter 6) is interesting when considered against the controversial remarks that sometimes erupt over mythicist publications. The code lays out the responsibilities of peer reviewers and researchers in Australian universities and other publicly funded research institutions. The mere fact that a code sets out responsibilities is itself testimony to the potential for the process to go awry. Its success relies upon the professionalism of enough of those involved to keep everyone in line and avoid lapses. The code is a valuable reminder of its potential limitations.
