Reporting negative results

Why Are Negative Results Still Six Feet Under?

Accueil Best practices Reporting negative results Why Are Negative Results Still Six Feet Under?

Table of content

1 – Introduction
2 – What is a negative result ?
3 – The shrinking space of negative results in scientific publications
4 – How to explain this under-representation?
5 – What are the consequences of this publication bias?
6 – What are the solutions?

I didn’t fail. I’ve just found 10 000 ways that won’t work.

Thomas Edison

By the end of the 19th century, Michelson and Morley tried to study the earth’s motion with respect to the luminiferous ether, a fluid supposed to fill all of space and act as a vehicle for light propagation. After many attempts carried out during several years, the two scientists never managed to prove their theory. Their work was however used several decades later as the basis of Albert Einstein’s theory of relativity, one of the most famous scientific discoveries.

Yet, negative results currently play a neglected role in academic publishing. Often considered as a burden for researchers’ careers, they are nevertheless  part of any scientist’s routine. “As widely recognised by philosophers of science, we can all learn from our mistakes, and errors can lead to discovery if they are properly diagnosed. However, failure stories are very
seldom communicated and published, even though they represent the bulk of the results obtained by researchers and modellers”, state several scientists in an article from a conference on hydrology organized in Paris [1].

In the current context of questioning the value of scientific publishing, scientific journals tend to be more open to negative results, adapting their editorial policies accordingly. We propose an overview of this issue, whose high scientific interest contrasts with its rather low position in institutional agenda. 

What is a negative result ?

The result of an experiment can be considered as “negative” or “null” when it does not support with sufficient statistical evidence the previously stated hypothesis. It does not necessarily mean failure as an unexpected outcome worthy of exploration might stem from it. Negative results are designated as such because they are to distinguish from positive results, which confirm the initial hypothesis.

However, the very concept of negative results lacks of a commonly accepted definition, often including a large set of notions. According to Rémi Thomasson, a biology oriented researcher and co-founder of the journal Negative Results [2], the latter can also refer to results conflicting with existing knowledge. This scenario appears when even after several attempts, a researcher is unable to reproduce an experiment in the scope of already published results. Those new results would in this way be in conflict with the initial state of knowledge and could therefore be qualified as negative.

The shrinking space of negative results in scientific publications

A 2012 study [3] brought to light the proportion of negative and positive results published in  scientific articles from 1990 to 2007. During this period, the so-called “positive results” increased by 20%. In 2007, they represented 86% of scholar publishing, at the expense of null or negative results, which kept losing ground.

This is a quite paradoxical assessment. A Spanish study [4] published in June 2021 tried to decipher the researchers’ point of view about the publication of negative results. Although 79% of the respondents do not seem to consider negative data as poor results and that 82% of them believe it should be widely spread within laboratories, only 14% of the respondents admit having ever tried to publish their results. How can we explain such a gap between the willingness of researchers to publish negative results and their effective low representation in the scientific literature?

How to explain this under-representation?

Several reasons can explain the poor status of negative results. The previous study also points out that the most important factor regarding the decline of published negative results is the lack of engagement amongst researchers. Devoting time to a task considered unrewarding naturally comes into conflict with the research activity. This phenomenon is called “publication bias” or also “file-drawer effect” [5]. The latter refers to the fact that negative results are often kept in researchers file-drawers, without ever giving them visibility.

The second factor is deeply-rooted in the scientific publishing industry: publishers tend to value cutting-edge discoveries likely to be highly cited. This tendency has two pernicious effects. First, it appears to restrict researchers freedom of publication. Second, the “publish or perish” principle in which negative results have no real place leads inevitably to ethical breaches [6]. Among them, we may refer to HARKing (Hypothesizing After the Results are Known) which consist in adopting an hypothesis after getting the experiment results, as well as p-value hacking, based on removing data going against the initial hypothesis.

Finally, the quality of negative results (as for any other type of data subject to bias) is facing a lot of challenges before one can definitely conclude that the experiment does not work: representative samples, a clear and well-detailed protocol, a rigorous calibration of instruments, significant statistical tests, etc. All of these parameters are of paramount importance not to produce false negatives. 

What are the consequences of this publication bias?

As previously stated, negative results tend to disappear from scientific literature, which alters the capacity of scientists to have a complete overview of their field of studies, as a huge part of knowledge, however “negative”, is considered as missing since it is not shared [7]. The situation is quite paradoxical as having access to complete view of any scientific field advancements would save time for researchers, exempted from repeating unnecessary experiments.

Are scholarly publishers likely to change their editorial policies? According to Bernard Rentier, former Liege University’s rector and author of Science ouverte, le défi de la transparence, there is only a few chances. “Interactions between researchers must therefore include all research achievements, whether glorious or not. Public platforms will only be able to publish negative results and it will be necessary to ensure that they do so effectively”. To date, only a few initatives have emerged. The associative project Gaffex is one of them, carried out by researcher Gilmary Gallon, working on setting up a platform for sharing “failed experiments”. A beta version is currently being developed.

What are the solutions?

In 1997, the Forum for Negative Results of the Journal of Universal Computer Science was created, allowing researchers to share and exchange their negative results in the field of computational sciences. Solutions to overcome the low visibility of negative results are slowly emerging within different scientific communities. Some publishers try to be game changers, as some scientific journals allow or even encourage the submission of negative results. 

  • Some scientific journals allow negative results to be highlighted :
    – For instance, a specific section in PlosONE, F1000Research, OSA Continuum and ACS Omega has been created for this purpose. A few years ago, publisher F1000Research called on researchers to publish their negative results in their journal. For a limited period, the journal waived the publication fee if researchers submitted papers based on unsuccessful experiments.
    – In October 2020, IOP announced a revamp of its editorial policy in favour of negative results in the journal IOP SciNotes.
    – PeerJ also allows the publication of negative results. Here is an example of an article that can be consulted here, with the comments of the three reviewers.
    – BMC research notes launched a collection dedicated to negative results.
    – The American Heart Association implemented a new collection named “Null Hypothesis Collection”.
    – Utrecht University launched the Journal of trial and error (JOTE) which is available in open access and without publication fees.
    – The Journal of Articles in Support of the Null Hypothesis is also a journal publishing negative results in psychology.
    – Another initiative comes from Emergent Scientist: this journal dedicated to young scientists’ comprises a dead-end section to share negative results in physics and mathematics.
    – Journal of Pharmaceutical Negative Results is also a journal focused on the publication of negative results. As its name indicates, this journal is oriented towards the publication of results in the pharmaceutical field.
    – Behavioral Neuroscience is a journal allowing negative results in the field of the neural bases of behavior.
  • Failed experiments can also be referenced within some bibliographic reference aggregators. This is the case of SciFinder-n, in chemistry, which gives the possibility to explore failed chemical reactions and to consult the causes in the comments when they are given.

To go further…

– True or False: Publishing Negative Results Ruins Your Science Career
– Be positive about negatives–recommendations for the publication of negative (or null) results
– Invisible science: publication of negative research results
– Highlight Negative Results to Improve Science
– It’s time for positive action on negative results

  1. Andréassian, Vazken, et al. « The Court of Miracles of Hydrology: can failure stories contribute to hydrological science? » Hydrological Sciences Journal, vol. 55, no 6, août 2010, p. 849‑56. Taylor and Francis+NEJM, doi:10.1080/02626667.2010.506050. Available: https://www.tandfonline.com/doi/pdf/10.1080/02626667.2010.506050?needAccess=true
  2. Mise en valeur des résultats négatifs
  3. Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904.
  4. Lucía Echevarría, Alberto Malerba, and Virginia Arechavala-Gomeza. Nucleic Acid Therapeutics. Jun 2021.185-189. http://doi.org/10.1089/nat.2020.0865
  5. Bernard R, Weissgerber TL, Bobrov E, Winham SJ, Dirnagl U, Riedel N. fiddle: a tool to combat publication bias by getting research out of the file drawer and into the scientific community. Clin Sci (Lond). Portland Press; 30 oct 2020;134(20):2729‑39 Available: https://portlandpress.com/clinsci/article/134/20/2729/226790
  6. Sharma H, Verma S. Is positive publication bias really a bias, or an intentionally created discrimination toward negative results? Saudi J Anaesth. 2019;13(4):352‑5. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6753760/
  7. Mlinarić A, Horvat M, Šupak Smolčić V. Dealing with the positive publication bias: Why you should really publish your negative results. Biochem Med (Zagreb) [Online]. 15 oct 2017 [cité le 16 nov 2020];27(3). Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5696751/