How junk science got spread like wildfire
The remarkable story of how scientists with some of the worlds biggest platforms shared a fatally flawed pre-print
In 1996, Professor Alan Sokal published an article called, “Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity” in the scientific journal “Social text”. Except the article was a hoax. In the words of the authors, it was.
“liberally salted with nonsense”
The aim was to see if such nonsense would get published if it sounded good and flattered the editors' ideological preconceptions.
Recently, a study was uploaded to the pre-print server medRxiv which was so riddled with basic errors that it led many of us to question whether this was an attempt at a similar style of experiment.
If it was, it certainly worked.
What is the study?
This pre-print set out to be a systematic review of research about the efficacy of facemasks for preventing Covid-19 transmission. Within less than 10 seconds of reading the abstract, it quickly becomes apparent that it falls well beneath the most basic standards of a systematic review.
It would take hours to go through every mistake in the paper, so I will quickly list the most egregious.
The whole purpose of a systematic review is that it should be a “systematic” search and synthesis of the published literature, and it should be easily replicated by reading the study methods. The paper gives zero- yes literally zero - details of how the papers were selected from the literature search, yielding a total of only 13 from over 1700.
Despite being uploaded in August 2022, the literature search is limited to between March and July 2020, more than 2 years ago. THOUSANDS of new papers will have been published on the topic since then, whose findings are ignored.
Despite the study period being between these 4 months in 2020, it somehow manages to include a study from 2004.
Despite being reportedly only about Covid-19, the same paper above is obviously not about Covid-19, but about SARS.
Despite one of the papers being described as being about use of facemasks on aeroplanes, the paper listed is actually about using powered, personal respirators when performing tracheostomies.
The manuscript provides no description for the methods of statistics used, other than they planned to use t-tests.
Several numbers in the study are not even consistent between the text, tables and the papers they are referencing.
The study reports a totally incomprehensible statistic of p value for the number of people wearing masks who got Covid-19, and another p value for the number of people not wearing masks who didn’t get Covid-19 (or SARS apparently…)
The study inappropriately adds totally incomparable numerators and denominators together from all the included studies, with no attempts to adjust for the extraordinary number of confounders, to produce its headline statistic.
To put it bluntly, the results of this study are of no use at all. It is completely uninformative and would not pass as an undergraduate medical student assignment.
None of these are high level, expert errors. These are the fundamentals. Anyone who claims to understand systematic reviews on any level should immediately be able to see that it falls short of the most basic scientific standards.
What happened next?
To the horror of many onlookers with an interest in Covid-19 science, this study spread like absolute wildfire from some of the biggest scientific accounts on twitter.
Not because it was bad.
Because they were sincerely using it as proof of the effectiveness of mask wearing.
We are not talking small fries here.
We are talking an ex-director of the World Health Organisation (90,000 followers), a professorial member of a UK based scientific political activist group (200,000 followers), a Pulitzer prize winning science journalist (260,000 followers), head of a translational research laboratory in the US (600,000 followers) and even the health minister for a large western European country (1,000,000 followers).
According to Altmetric, the pre-print has also been cited in 10 news articles - all of which using it as evidence of the effectiveness of masking.
How has this happened?
There are only 2 explanations for how such senior scientists could end up in this situation:
They are unable to perform basic critical appraisal of systematic reviews
They shared the study having not read it at all
Realistically (and hopefully…) point number 2 is the most likely. In one interesting twitter conversation, a non-medical scientist responded that they had shared it because it had been shared by a different professor, and that was good enough endorsement for them
The original professor then said they had shared it because it had been shared by a Pulitzer prize winning journalist, and that their sharing it again was not an endorsement. They did not comment any further on whether they had actually read the study or whether it was appropriate to share such a study which lacked rudimentary scientific legitimacy.
This appears, for the most part, to have been like a bad game of Chinese whispers.
Someone tweeted a very, very bad study, presumably without reading it.
Someone else liked the results and likes/trusts the original tweeter, so re-tweets it (to millions of people) without reading it.
Someone else liked the results and likes/trusts the original tweeter, so re-tweets it and so on and so on….
Confirmation bias is such a powerful motivator that the desire to smash “Retweet” when seeing this kind of result which aligns with your preconceptions is almost irresistible.
Even if you don’t have the time or inclination to read it.
This force is even stronger when the topic is so ideologically divided, such as in the case of facemasks. As a result, it is likely that tens of millions of people have seen this study promoted as if it provides a legitimate estimate of the effectiveness of facemasks for Covid-19. Please recall, we are discussing a systematic review that has accidentally included a study from 2004 about SARS and doesn’t even have consistent numbers in it.
Conclusion
If you are a senior scientist with a platform of hundreds of thousands, or even millions of followers, you cannot simply amplify studies because you like the results and someone you respect has already shared it.
People put their trust in scientists to be thorough and objective if nothing else. It is absolutely no excuse to say,
“Someone I respect shared it and sharing it doesn’t mean I am endorsing it”.
We all have a huge responsibility for what we put out in public facing communications, as it impacts not just peoples understanding of science, but their trust in the scientific establishment. This should be true of all scientists, but those with large social media following must take into account that this influence brings even more responsibility.
If people can’t trust us to have even performed the most basic quality check of research before we boom it out to millions of people, how can they trust us with anything more complex?
Alas, it seems it is possible to get your study amplified even if it is “liberally salted with nonsense”, so long as it flatters peoples ideological preconceptions.
Alan Sokal wouldn’t be surprised.
I really want to share this post, but am I now intellectually obliged to read the medRxiv paper in question to check it is as you describe?
This is unacceptable behavior by scientists! May I also add to your report reason number three? That would be that even though they knew it was filled with errors they went ahead and accepted it to support their greedy and controlling narrative. And they banked on most folk’s short attention span and lack of knowledge.
Thank you for sharing this! And for researching it so thoroughly and explaining it so well!