Category Archives: Peer Review


2013-10-21

Peer Review — current developments

by Neil Godfrey

Sian Harris has made some interesting comments about current trends in scholarly publishing, including observations of what is happening to peer-review.

On peer-review (my bolding) one of the developments noted is:

Peer review is another interesting trend to watch. Different journals take different approaches to this. One trend is from blind to open peer review, where authors and reviewers know each other’s identity. Another topic for discussion is whether the lion’s share of peer review should go on pre-publication or post-publication. The journal PLOS One has an interesting approach to this, of deliberately only assessing papers for things like originality, accuracy and ethics but not making a judgement on how interesting the research is before publication. The discussion of the value of the research goes on afterwards.

There’s a related article, New Approach to Peer Review.

(Not that any of this is necessary for the Humanities, according to Larry Hurtado. Everything is just fine there. It’s only where those scientists have the ability to fabricate data that we find any problems, according to his ostrich perspective.)

 


2013-10-15

If Peer-Review Does Not Work for Science Why Does It Work for Biblical Studies?

by Neil Godfrey
quote_begin

We have little or no evidence that peer review ‘works,’ but we have lots of evidence of its downside.

quote_end
Morgan as The Gatekeeper at the entrance to th...

The Gatekeeper at the entrance to the Emerald City (Photo credit: Wikipedia)

Mythicists have been told so often (oh how so very often) that they should publish in peer-review journals to be taken seriously. Peer-review, the public has been repeatedly told, is the guarantee of true scholarship. In the recent scholarly outrage over Joseph Atwill’s thesis gaining more public attention than the works of the academy, Larry Hurtado reminded us that “peer-review” and “reputable” go together like carrots and peas and Tom Verenna was once again extolling peer-review as the magic gateway guaranteed to keep intellectuals worthy and honest.

Now I do understand the reasons for peer-review. But what these reminders from among biblical scholars are overlooking is the research that demonstrates that peer-review in the various ways it is practiced today is also a deeply flawed process. Or maybe all those studies demonstrating this have no relevance for theologians.

You see, there is a conflict between what I read on the web by Bible scholars about the how effective the peer-review process is in their profession on the one hand, and what I read about the flaws in the peer-review process in my professional capacity (coordinator of a research data management project and of a research publications archiving and access project) in an academic institution on the other.

While I read of the virtues of peer-review for maintaining the pure standards of biblical scholars after hours, during my work time I am reading published research findings that are not so sanguine about peer-review. Why the difference? Could it be that the research is focused mostly in the areas of the sciences. No doubt the nature of that sort of material makes objective analysis easier. Does that mean the demonstrated failings of peer-review could never apply to the field of biblical studies?

Given that scientists are increasingly being exposed to an understanding of the flaws in the peer-review process, are we to assume that biblical scholars are immune from these flaws and that their peer-review mechanisms really are guarantors of quality work?

One article that referenced several studies on the peer-review process is Richard Smith’s “Classical peer review: an empty gun” in Breast Cancer Research 2010, 12(Suppl 4):S13 doi:10.1186/bcr2742 (a peer-reviewed journal).

If peer review was a drug it would never be allowed onto the market

This is how the article begins. It is a quotation from the deputy editor of a leading medical journal and “intellectual father of the international congresses of peer review that have been held every four years since 1989”, Drummond Rennie.

Later the article makes this claim:

If peer review is to be thought of primarily as a quality assurance method, then sadly we have lots of evidence of its failures. The pretentiously named medical literature is shot through with poor studies.

One would think that in a field like medical studies that peer-review would ensure that only accurate information is published. Certainly we would not think that the peer-review process would let through anything that would cause public harm.

But the facts prove otherwise.

There is much that is published that is downright false. The editors of the ACP Journal Club find that less than 1% of studies in most journals are “both scientifically sound and important for clinicians”. There are also documented instances of bad studies being published that have led to patient heart attacks and measles epidemics.

Note the following and ask if we have the same types of human nature producing and reviewing articles in biblical studies (with my bolding and formatting):

Doug Altman, perhaps the leading expert on statistics in medical journals, sums it up thus: ‘What should we think about researchers who

  • use the wrong techniques (either wilfully or in ignorance),
  • use the right techniques wrongly,
  • misinterpret their results,
  • report their results selectively,
  • cite the literature selectively,
  • and draw unjustified conclusions?’

We should be appalled. Yet numerous studies of the medical literature have shown that all of the above phenomena are common. This is surely a scandal.’

Back to Drummond Rennie:

Drummond Rennie writes in what might be the greatest sentence ever published in a medical journal:

DrummondRennie

D. Rennie

‘There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.’

Are biblical scholars more professional as a whole than doctors? Are their arguments and publications more rigorous? According to everything I read by biblical scholars themselves I must think they really are. No doubt our souls are worth much more care than our physical bodies.

Richard Smith continues:

We have little or no evidence that peer review ‘works,’ but we have lots of evidence of its downside.

What are the downsides? read more »