Photo by Kirk Durston

The primary way scientific discoveries and advances are disseminated is through peer-reviewed papers published in scientific journals. The first step is to submit a paper to a journal. Those that survive preliminary filtering by the editor are sent out to be reviewed by qualified scientists in the field. On the basis of the reviewers’ recommendations, the paper is accepted or rejected. Only a fraction of papers submitted for publication make it through this peer-review process and are published.

One would hope that such a process would justify a high level of confidence in scientific publications, but recent findings suggest that our faith in peer-reviewed publications in mainstream journals of science may be on somewhat shaky ground.

The journal Nature, in a paper calling for increased standards in pre-clinical research, revealed that out of 53 papers presenting ‘landmark’ published findings in the field of haematology and oncology, only 6 could be confirmed by subsequent laboratory teams. For the 89% of papers that failed to have their results reproduced, it was found that blind control group analyses was inadequate or data had been selected to support the hypothesis and other data set aside.

Worse still, some of the papers that could not be experimentally reproduced, launched ‘an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis’.

It is surprising that “hundreds” of other peer-reviewed, published science papers were based upon faulty initial papers but it occurred.

Nature reported in October 2011 that although the number of submissions had increased by 44% over the past ten years, the number of retractions had increased by roughly 900%.

Austin Hughes, in a paper published in the Proceedings of the National Academy of Sciences, focusing on the origin of adaptive phenotypes laments, ‘Thousands of papers are published every year claiming evidence of adaptive evolution on the basis of computational analyses alone, with no evidence whatsoever regarding the phenotypic effects of allegedly adaptive mutations.’ He concludes that ‘This vast outpouring of pseudo-Darwinian hype has been genuinely harmful to the credibility of evolutionary biology as a science.’ Jerry Fodor and Massimo Piattelli-Palmarini write in New Scientist,

“Much of the vast neo-Darwinian literature is distressingly uncritical. The possibility that anything is seriously amiss with Darwin’s account of evolution is hardly considered. … The methodological skepticism that characterizes most areas of scientific discourse seems strikingly absent when Darwinism is the topic.”

How can we distinguish the good papers from the poor? This can be very difficult without actually attempting to reproduce their findings. Short of that, apply the same critical thinking skills and healthy skepticism to scientific papers that you do for political, historical or religious claims. 21st century science can often be heavily influenced by poor experimental practices, questionable statistical sampling and analyses, unproven computational models, political agendas, competition for funding, and scientism (atheism dressed up as science). When going over a paper ask questions like, how large was the data set? What sort of statistical analysis was performed? Are there other papers that independently support or disconfirm these findings? What is not being discussed? One thing for sure, don’t accept something simply because ‘hundreds’ or even ‘thousands’ of papers say so, especially if Darwinian evolution is the topic and reproducing the experiment is not possible. Practice critical thinking with the question in the back of your mind, ‘Is this one of those papers that will be retracted?’.

Read Part III

Further reading:

  1. ”Severe’ figure manipulation found in studies from leading plant lab’Nature, Sept. 2018.
  2. ‘What is threatening science?’Project Syndicate, by Jeremy Baumberg (nano-scientist, University of Cambridge), Sept. 2018.
  3. ‘Academic Activists Send a Published Paper Down the Memory Hole’Quillette, Sept. 2018. See also ‘A twice-retracted paper on sex differences ignites debate’, Sept. 2018.
  4. JAMA Journals Retract Six Papers by Cornell Researcher, The Scientist, Sept. 2018.
  5. Sabine Hossendfelder, ‘Science has a problem, and we must talk about it’, Sept 2018.
  6. ‘No more excuses for non-reproducible methods’Nature, 2018.
  7. ‘Replication Failures Highlight Biases in Ecology and Evolution Science’The Scientist, August, 2018.
  8. ‘35,000 papers may need to be retracted for image doctoring, says new paper’Retraction Watch, June 2018
  9. ‘Scientists Rarely Admit Mistakes. A New Project Wants to Change That’UNDARK, July, 2018.
  10. ‘Opinion: We Must Demand Evidence of Peer Review’The Scientist, May 2018.
  11. ‘Nature says it wants to publish replication attempts. So what happened when a group of authors submitted one to Nature Neuroscience?’ A fascinating, three-part, behind-the-scenes account of how Nature Neuroscience, refused to publish a paper that reported the failure to replicate a landmark paper that had previously been published by Nature Neuroscience, despite the fact that the attempt to replicate the paper was performed by 9 labs, Retraction Watch, May 2018.
  12. ‘Recognizing “Spin” in the Scientific Literature’PLOS Blogs, May 2018.
  13. ‘NAS Launches New Report: “The Irreproducibility Crisis of Modern Science” ‘National Association of Scholars, April 2018.
  14. ‘Scientists aim to pull peer review out of the 17th century’Shots:Health News from NPR, Feb 2018.
  15. ‘We should not accept scientific results that have not been repeated’Nautilus, Feb 2018.
  16. ‘Best-selling introductory Psychology books give a misleading view of intelligence’, The British Psychological Society Research Digest, March 2018.
  17. ‘Prestigious Science Journals Struggle to Reach Even Average Reliability’Frontiers in Human Neuroscience, February, 2018.
  18. ‘Most insect studies lack crucial species information’Nature, (2018)
  19. ‘Definitely embarrassing”: Nobel laureate retracts non-reproducible paper in Nature journal’, Retraction Watch, December 2017
  20. ‘Spin’ in published biomedical literature: A methodological systematic reviewPLOS Biology, Sept. 2017
  21. ‘Science spin prevalent, researchers warn: Extent of spin in biomedical papers revealed’, Science Daily, Sept. 2017
  22. Sabine Hossenfelder, ‘Research perversions are spreading. You will not like the proposed solution’, Backreaction, December, 2017.
  23. ‘Peer-review fraud scheme uncovered in China’The Scientist, 2017
  24. Metascience: Reproducibility Blues‘, Nature, March, 2017
  25. ‘Quackery infiltrates the BMJ’Science-Based Medicine, May 2017
  26. ‘Publish houses of brick, not mansions of straw’Nature, May 2017
  27. ‘Widely-reported study on fish and microbeads might have been faked’, Gismodo, 2017
  28. ‘National academies release sweeping review of research misconduct’, Physics Today, April 2017
  29. Here is a “sting” exposing the practices of a particular category of science journals classified as “predatory” (exist for making money from authors, rather than promote good science–where the ‘review’ consists mainly of whether one is willing to pay the fee to have their paper published), ‘Predatory Journals Recruit Fake Editor’Nature, March, 2017.
  30. ‘Ranking major and minor research misbehaviors: results from a survey among participants of four World Conferences on Research Integrity’Research Integrity and Peer Review, Nov. 2016
  31. ‘Bee Experts Challenge Environmental Claim that Wild Bees are Near Extinction’, Science 2.0, April, 2017
  32. ‘Fake Research’ Comes Under Scrutiny, BBC News, March 2017
  33. ‘Science, lies, and video-taped experiments’, Nature, Feb. 2017
  34. 80% of Chinese Clinical Trials Data Fabricated, Pharmafile, September, 2016
  35. 1,500 scientists lift the lid on reproducibility, Nature, May 2016.
  36. ‘Offline: What is Medicine’s 5-sigma?’The Lancet, 2015.
  37. Why Most Published Research Findings are False, PLOS Medicine, 2005
  38. A Dig Through Old Files Reminds Me Why I’m So Critical of Science, Sci Am, 2013
  39. Author with Seven Retractions makes Thomson-Reuters List of Top Scientists-Plus Another Twist
  40. In Science, is honesty always the best policy?
  41. ‘Bad Science’ConservationBytes, February 2016.
  42. How scientists fool themselves-and how they can stop
  43. Fraudulent Paper pulled
  44. Drug development: Raise standards for preclinical cancer research
  45. Publishing: The peer-review scam
  46. A sharp rise in retractions props calls for reform
  47. Survival of the fittest theory: Darwinism’s limits
  48. Science, now under scrutiny itself
  49. 64 more papers retracted for fake reviews, this time from Springer Journals
  50. Study: Peer Reviewers Swayed by PrestigeThe Scientist, Sept. 2016
  51. Psychologists fail to replicate well-known behaviour linked to learning, Nature, Sept. 2016

Subscribe to receive Contemplations straight to your inbox!

Email Address:

Share This