Measuring what matters
Surrogates vs clinical outcomes in immunity to Covid-19
After a recent post of mine on twitter that both vaccination and infection for Covid-19 provide robust protection against reinfection, someone responded;
“Infection provides a much weaker response”
I disagreed, and provided a number of studies which demonstrate significantly reduced rates of reinfection, severe disease and death following infection which are comparable, or in some studies superior to vaccination (for a list of references with links see footnote)1.
The response was;
“You posted a thread with several observational studies that do not (and cannot) say anything about the relative strength or protection of the immune response.”
This is an astonishing statement, but I’m afraid stems from a common misunderstanding.
Surrogate vs Clinical Outcomes
The argument from my challenger was that in laboratory studies, the responses to vaccination appear better than responses to infection. This may well be true. But laboratory measurements of what cells are doing is not ultimately what we care about.
Imagine a new cancer treatment which in a lab is shown to kill cancer cells highly effectively, and when given to patients its shown to shrink tumours by 50% in a matter of weeks. A success!
However, the patients do not live any longer when given the treatment, and do not feel any better. A categorical failure.
The things that matter in medicine are the clinically important outcomes that happen to real people.
This is regardless of what the laboratory or mechanistic evidence shows. This is the reason why endless therapies which succeed in the lab ultimately fail when they reach clinical trials. We can try to make educated guesses about what might happen clinically based on laboratory data, but what happens in the lab is irrelevant unless it is matched by meaningful changes to clinical outcomes.
We’ve seen this before in the case of immunity against Covid-19. Remember the breathless newspaper articles claiming Omicron infection provided little to no protection against reinfection based on mechanistic laboratory evidence of immunity?
Turns out that was completely wrong, as comprehensively demonstrated by the observational data in real people.
The observational evidence is clear that immunity against Covid-19 conferred by infection or vaccination are comparable (and protection from infection may be more durable).
In fact, these studies are the only way of demonstrating how strong immune protection is following infection (randomised trials would be better evidence, but we cannot randomise people to getting infected). Protection from immunity cannot be demonstrated in a lab. Protection can only be measured in clinical outcomes, such as hospitalisation, death, or symptomatic infection.
That is not to say that getting infected is desirable - it is the very thing you are trying to protect yourself against! Immunity from vaccination is overwhelmingly safer for the vast majority of people. But if you are to be infected, it is reassuring to know that afterwards you have increased protection against reinfection.
What about bias?
Some might try to dismiss the findings of these observational studies as simply being prone to bias. Easy to say, but much less easy to justify. For one thing, we know observational studies of vaccine effectiveness are reliable, because their results very closely resemble the findings of randomised, controlled trials - the gold standard of evidence.
One concern is that these studies could be influenced by test seeking behaviour (detection bias) - however a much more detailed study of this nature from the Netherlands found that this could only account for a small fraction of the observed effectiveness at most.
In addition, these biases can only affect the outcome of symptomatic infection. Hard outcomes such as hospitalisation and death are not easily biased (you cannot choose to die or not).
There are potential confounders which could influence differences in these outcomes, but these are more likely to influence comparison between vaccinated and unvaccinated people. When comparing infected vs uninfected people, bias is more likely in the other direction (those who have already been infected will represent a population higher risk of getting infected; hence why they got infected). The other major confounders in risk of severe illness are much easier to control for, such as age and presence of comorbidities.
Finally, there are UK based studies using mass population testing (via the ONS infection survey) that are free from almost all these issues and find precisely the same result (see Table 2: Pouwels et.al. Nature Medicine, 2021).
Summary
Laboratory evidence is vital to understanding how and why things happen in medicine. However, ultimately the outcomes we care about are the clinical outcomes that occur in human beings. Observed outcomes in laboratories are only surrogates for the outcomes that really matter, such as feeling well, avoiding morbidity and living longer. For Covid-19 immunity, these show that immunity following infection is comparable to that following vaccination.
For the second time it pains me to quote Rob Hughes when he said;
Measurement eats mechanism for breakfast.
References on immunity following infection vs vaccination
Pouwels et al. Nature Medicine, 2021
Chemaitelly et al. Lancet microbe, 2022
Andeweg et al. Nature Communications, 2022
Thank you for patiently explaining this! Having serious discussion on Twitter is pretty tough.
Of course surrogate endpoints are used in cancer clinical trials all the time (generally progression-free survival). This has been pointed out as a troubling trend.