Literature reviews can help to synthesise a lot of information
There is an enormous amount of academic research in the world. And the number is growing all the time. The volume of research articles doubles every nine years. In 2010 there were already estimated to be more than 50 million research papers on pretty much every subject imaginable (and some we’d struggle to imagine).
In any one topic area there’s often so much research that people wanting to rapidly understand a subject and make a decision for policy or practice cannot read up on it comprehensively.
This is one of the main reasons that researchers write literature reviews. They provide a summary of a complex, large evidence base in a way that can be quickly and easily understood.
Some literature reviews act as introductions to a well understood topic. Some aim to summarise key theories that explain a phenomenon. Some try to comprehensively map what research has been done and how. Others aim to accurately and precisely summarise the impact of one factor on a specific outcome.
In many cases reviewers try to be comprehensive or transparent so their work could be repeated by other scientists and their conclusions can be verified relative to the evidence they found.
But there’s a problem. Reviewers might reach a conclusion that uses unreliable evidence or summarises the evidence in a way that isn’t as reliable as it could be. This might cause policymakers or practitioners to make an expensive or damaging decision.
One well known example is a review published in 1967 that suggested fat and cholesterol – not sugar – were the most important dietary factors in causing coronary heart disease. The review was funded by the sugar industry, which also contributed to the planning and conduct of the review, even though no details of this were reported.
A detailed recent analysis showed the review used unreliable and biased methods to claim that research linking sugar to coronary heart disease was invalid. The original finding benefited the sugar industry, but it may have caused significant damage to human health because of ill-informed policy.
This is why researchers should try to make their literature reviews as reliable as possible. They can do this by following guidelines and standards when they plan and conduct their reviews. Another approach is to adhere to established standards in how to report what they did and found in detail.
That’s what prompted myself and several colleagues to design a new set of rigorous standards specifically for reviews that do not fit the well defined field of health care.
Such standards help review authors to remember to report all of their methods in sufficient detail to be repeatable and verifiable. In the “real world”, this means that policymakers and practitioners can make decisions based on the best available evidence, helping to ensure their actions are likely to succeed.
Transparency issues
The methods that a literature review employs – or doesn’t – affect how reliable the review is as a whole. If a review skips an important step, the review conclusions may not be correct.
Generally speaking, two key factors affect the reliability of a literature review.
One is the quality and appropriateness of the studies going into the review. The other is the quality and appropriateness of the methods used to review the evidence. There’s a whole suite of carefully designed literature review methods to ensure these factors are addressed.
In describing their methods, authors should outline the system they studied, how they measured what they were interested in and how they calculated their summary data in detail so that someone could repeat exactly what they did. This allows the reader to be sure that the author has acted appropriately with the available information.
Instead, literature reviews very often lack even basic transparency regarding how they were undertaken and therefore cannot be repeated or checked for errors. Improving transparency shouldn’t be an onerous task: reviewers will know exactly what they have done and will probably have computerised records that outline their process and methodology.
Implementing new standards
There are some established, tested standards to help review authors be more transparent and reliable. Preferred Reporting Items for Systematic reviews and Meta-Analyses, or PRISMA, is commonly used in the field of health care.
Recently, I worked together with Biljana Macura (Stockholm Environment Institute), Paul Whaley (Lancaster Environment Centre), and Andrew Pullin (Bangor University) to develop a new set of standards. It’s called Reporting standards for Systematic Evidence Syntheses, or ROSES and was produced specifically for those reviews that do not fit the well-defined field of health care.
These standards help review authors to make sure they have included all relevant information in their review reports. They also help authors to design their reviews from the start, and then help journal peer reviewers and editors to better understand how well the review was conducted.
Several journals, including Environmental Evidence, Environment International and Nature Climate Change have begun to integrate ROSES forms into their publishing work flows. Their hope is that review authors will produce and submit more transparent and reliable reviews.
By widely adopting these reporting standards across academic journals, the research community can become more aware of the importance of transparency and clarity in how literature reviews are described.
This can help decision makers to identify when a review is reliable or not, and focus on the most reliable evidence when they are making decisions.
Neal Robert Haddaway, Research Fellow, Africa Centre for Evidence, University of Johannesburg
This article was originally published on The Conversation. Read the original article.
