Z is for Zombie

Zombie statistics are figures which are understood to be true, although they have no or little basis in fact, and just won’t die.

Animal research discussions often reference the statistic ‘90% of drugs tested on animals fail’ (or similar) to argue for or against animal research. However, it is often difficult to find the source, or sources, of this figure and how exactly it was calculated. Peta, the People for the Ethical Treatment of Animals, say it comes from the British Medical Journal (BMJ) in a 2014 review: '“one study found that fewer than 10% of highly promising basic science discoveries enter routine clinical use within 20 years.” As Peta didn’t link to the review, it takes a little digging to find the reference, which in turn references other research papers.

In the other corner, the pro-animal research group Speaking of Research say it comes from the US Food and Drug Administration in 2006, however their reference is a quote, not a research article, so it holds little weight when it comes to evidence: “Health and Human Services Secretary Mike Leavitt stated that “Currently, nine out of ten experimental drugs fail in clinical studies because we cannot accurately predict how they will behave in people based on laboratory and animal studies.” This provides a little more context, but their links lead to 404 pages so the trail goes cold.

While still promising and not yet proven to be a zombie statistic, both of these examples display sloppy, at best, referencing and support of their respective cases as both have essentially begun their arguments with ‘my friend said…’.

The trail is still warm for Peta, and their BMJ review cites a 2003 review in The American Journal of Medicine (AJM), ‘Translation of highly promising basic science research into clinical applications’. The paper’s discussion finds that “fewer than one in 10 [promising technologies] entered routine clinical use within 20 years of the index basic science publication”. However, the review restricted their research to 101 articles published between 1979 and 1983 and only those they “identified” from “six major basic science journals” in which the research hadn’t yet led to regular medical use.

Further searches found many articles without links or sources.

Just because a government institute of health makes a claim, it doesn’t mean it is true. Of those who did include a link or source, the articles cited and referenced each other ultimately leading back to the 2003 AJM article. Others concluded that 90% or ‘9 in 10’ drugs failed but decided that this figure could be calculated by adding together the ~30% of drugs that fail to progress from phase II drug development studies with the ~58% that fail in phase III. Additionally, they fail to add to the context that animals only make up a part of these phases. Animal research advocates Understanding Animal Research add, “Percentages thus become less and less helpful for understanding what is happening.”

As we would criticise animal researchers for failing to do, neglecting to encapsulate the entire context of a figure, particularly one which is so widely used to attack and discredit researchers, removes all meaning and integrity from those shouting the figures from their street corners.

The cost of developing new drugs is also often scrutinised and reported by detractors to be somewhere around $US1-2 billion. A study from the Journal of the American Medical Association in 2020 found that a more accurate median research and development cost of new drugs, including expenditures on failed trials, approved by the US Food and Drug Administration between 2009 and 2018 was $US985 million.

While far more accurate than the failure percentage rate, these figures still lack context. To arrive at this cost the study considered other financial amounts not directly related to the laboratory research, such as licensing fees and the cost of capital (return rate for investors). There are further issues with these figures in that they were only “based on data for 106 products developed by 10 large firms”, and on “costs voluntarily submitted by anonymous companies without independent verification, making them difficult to validate”. A small sample size, self-reporting, and unverifiable figures are the definition of poor science.

As Hilda Bastian concludes: “Just because a dramatic zombie statistic draws attention in a good cause, doesn’t excuse its falseness.”

Update, May 2024: Two new and prominent articles have appeared, claiming there is ‘updated’ information available on the 90% failure statistic. World Animal Foundation have updated their page on Animal Testing Statistics by revising the figure to 95% of drugs fail. They quote the British Medical Journal, however they link to a Peta page which links to another Peta page, then another, then another, before finally making its way to a National Institute of Health page on New Therapeutic Uses which make the claim without context or sources.

(We have contacted the NIH for comment on this figure and its use by animal advocate groups.)

The other article, from Sentient Media, claims “90 of drugs fail once they reach the human clinical trial stage” and briefly include context by adding, “There are a variety of reasons why this happens…”, before focusing on the use of animals in research without exploring the many other reasons scientific research may fail. Following the link of their reference leads to a Conversation article - 90% of drugs fail clinical trials - which links to subsequent articles explaining some of the many reasons, including patient selection, clinical trial complexity, lack of funding, and commercial decisions. Sentient Media seems to have deliberately misinterpreted the figure by omitting discussion about these issues. They add the claim that “animal testing is also expensive” by citing an article which, again, explains that much of the cost of research comes from “capital costs” and other “business reasons”.

In the pursuit of scientific advancement, failures are inevitable milestones on the road to discovery. They represent moments of learning, refinement, and growth. However, amidst our quest for knowledge, we must never lose sight of our moral compass. We must adamantly reject any notion that justifies the exploitation or harm of animals in the name of progress.

Sources:

Contopoulos-Ioannidis DG, Ntzani EE, Ioannidis JPA. Translation of highly promising basic science research into clinical applications. Am J Med 2003;114:477-84

Pound P, Bracken M B. Is animal research sufficiently evidence based to be a cornerstone of biomedical research? BMJ 2014; 348 :g3387 doi:10.1136/bmj.g3387

Van Norman GA. Phase II Trials in Drug Development and Adaptive Trial Design. JACC Basic Transl Sci. 2019 Jun 24;4(3):428-437. doi: 10.1016/j.jacbts.2019.02.005. PMID: 31312766; PMCID: PMC6609997.

Wouters OJ, McKee M, Luyten J. Estimated Research and Development Investment Needed to Bring a New Medicine to Market, 2009-2018. JAMA. 2020 Mar 3;323(9):844-853. doi: 10.1001/jama.2020.1166. Erratum in: JAMA. 2022 Sep 20;328(11):1110. Erratum in: JAMA. 2022 Sep 20;328(11):1111. PMID: 32125404; PMCID: PMC7054832.