The COVID-19 pandemic has resulted in an urgent need for scientific research to help us understand the reality of this virus and its toll on humanity. This demand for access to the latest studies has resulted in over 7,000 scientific articles being published in the past three months alone.
Having relevant public health information and access to the latest research can be vital to clinicians, especially in the midst of a pandemic.
The increase in publications creates a treasure trove of resources for us to explore. However, we have to do our due diligence to ensure that we are using reliable and scientifically accurate sources to cite.
Is fast-tracked COVID-19 research reliable?
Many of the recent research articles on COVID-19 aren’t going through the rigorous standards that publishers put in place previously to weed out shoddy science and poor methodology.
Traditionally, prior to publication, researchers had to submit articles to a team of experts who would, ideally, review the science and question the authors about their work. Many of the published COVID-19 papers are not undergoing this process and are instead being released through preprint servers.
To learn more about this issue, I spoke with Dr Ivan Oransky. He’s a physician turned journalist who co-founded Retraction Watch, a blog that keeps track of retracted scientific articles.
“During a pandemic, you can understand why people want studies to get out there more quickly. They want other scientists to build on their results,” he says.
While preprint servers have been around for nearly thirty years, the current main preprint servers for medical and scientific papers, MedRxiv and BioRxiv are relatively new. These servers publish studies that have yet to be peer-reviewed or extensively edited.
Dr Oransk continues: “The risk, of course, is that people don’t see the difference between a preprint and a published paper, because they kind of look the same— at least to an untrained eye. Anyone can publish a preprint without any training.”
This leaves the reader with the responsibility of questioning the science.
COVID-19 retractions
Retraction Watch’s page on COVID-19 retractions lists 15 retractions of papers published so far. The majority of these papers were published in the last two months.
According to Dr Oransky: “On average retractions take about three years. These (seven retractions) are taking days or weeks at most.”
Does this mean that health writers shouldn’t source preprint papers? Not necessarily, as long as the writer notes what kind of paper it is.
“If a manuscript is stated not to have undergone peer review, a health writer should state this when citing the paper,” says Dr Yasmine Ali, Assistant Clinical Professor of Medicine at Vanderbilt University and the President of LastSky Writing, LLC.
Simply put, problematic scientific articles can be found in all publications not just preprint papers.
“Frankly, our experience at Retraction Watch shows that plenty of really dodgy science gets through journals too and gets through peer review. It’s a complex issue,” Dr Oransky states.
The main takeaway is that health writers should remain vigilant in verifying the validity of a source (peer-reviewed or otherwise) before citing it. Here are some tips to help with that.
Check out the journal
Not all journals are equal. Unfortunately, there are journals that ask authors to pay a fee in order to get published. These predatory journals will publish questionable research and take advantage of readers and authors alike.
Other journals have a vigorous system in place to screen incoming scientific research.
“Look specifically for the manuscripts that are in journals with high impact factors. These are much likelier to be higher-quality research papers due to the integrity of these publications and their internal editing processes,” Dr Ali advises.
She suggests searching for the impact factor on a search engine and to take into consideration that each medical specialty will have its own set of journals.
Some of the examples she gave of high-quality journals are The Lancet, New England Journal of Medicine, British Medical Journal, Nature and Science. “The higher the impact factor, the more reliable the journal,” she says.
Double-check the author’s credentials
Scientists and researchers are required to list all their credentials. Health writers have to take the extra step to look into these author credentials.
This information should be easily available with a simple online search to verify that the authors do have the qualifications listed. There have been cases of unknown or missing authors on questionable research papers.
A quick look at any record of previously retracted papers is also a good idea. Retraction Watch is one resource you can use, but a quick search online can also reveal a lot.
Don’t just read the abstract: analyse the article
Abstracts are meant to summarise the research and hit on the key findings. They are not meant to replace reading the entire article.
Always read the entire article to get a better grasp of the science. When reading through an article, here are some questions you should be asking yourself.
- Are the methods clearly explained and capable of being reproduced?
- Are the figures and tables clear and easy to understand?
- Do they line up with what is written in the results section?
- Is there a limitations section?
- Do researchers explain how their results still hold up despite the mentioned limitations?
- For a clinical study, was there a control group? Or were all participants given the treatment/drug?
- Are researchers only relying on case reports for their results?
Verify the research with experts
As health writers, we may find ourselves writing about a broad range of topics that are outside of our personal knowledge. We don’t have to know it all, but we do have to ensure that what we write is factual. It’s important to reach out and seek expert opinions.
Find experts in the field and ask them about their thoughts on a specific manuscript.
Don’t feel afraid of appearing unintelligent or asking stupid questions. If something doesn’t make sense to you, there’s a good chance you aren’t the only one who thinks so.
Experts are also clued into their respective academic or scientific communities. They are more likely to know which researchers or scholarly works are problematic or representing a conflict of interest.
Single studies vs bodies of science
It seems every couple of days a new drug or treatment is being touted as effectively treating COVID-19 only to be later refuted by another paper. (Take the recent hydroxychloroquine controversy for example).
“In a perfect world, I would… all together stay away from single papers because more often than not they don’t hold up or they turn out to be an incomplete version of reality,” Dr Oransky says.
Multiple studies finding similar results with a large number of patients hold more weight than a single study showing surprising results. However, it can be argued during a pandemic any study that shows promising results should be shared. It’s relevant and newsworthy.
So, what should a health writer do? Try to focus on the bigger picture. How does this one study compare with the literature? Refer back to the methods used and be sure to include any limitations such as a small number of patients or the lack of a control group.
“It’s much more important to include context and try and report on the state of the science rather than the state of a single study. That’s challenging particularly in the time of a pandemic,” Dr Oransky states.
This post hoped to provide tips for health writers on how to discern whether a published work is a reliable source. In light of all the research being shared with the world at lightning speed, we need to step up as health communicators and do our part in sharing good science and calling out misinformation. We serve an important role in sharing and explaining complicated science to the general public. It’s our responsibility to ensure that the information we are sharing is accurate.
