The Importance of Correct Infection and Exposure Pool Estimations when Making a Comparison Between COVID-19 Vaccine Injury Rates Among Vaccinees and COVID-19 Injury Rates Among Unvaccinated Individuals

by Anette Stahel, MSc

Summary

On August 25, 2021 the above paper, Safety of the BNT162b2 mRNA COVID-19 Vaccine in a Nationwide Setting, was published in New England Journal of Medicine. Unfortunately, the study includes two method errors which make the comparison between COVID-19 vaccine injury rates and COVID-19 injury rates in it incorrect. More specifically, the vaccine injury rates among vaccinees are compared to disease injury rates among confirmed infected people when instead they should be compared to disease injury rates among the total pool of unvaccinated people. I'll here explain how come using highly adequate infection and exposure pool estimations when conducting such a comparative study is of utmost importance. I'll also carry out a more correct calculation of the total pool of unvaccinated people, based on official infection rate figures. 

Introduction

On August 25, 2021 the above paper, Safety of the BNT162b2 mRNA COVID-19 Vaccine in a Nationwide Setting, authored by Barda et al from the Israeli Clalit Research Institute (CRI), was published in New England Journal of Medicine [1]. The title of the paper describes its content very well, although in addition to investigating the occurence of various injuries following BNT162b2 mRNA COVID-19 vaccination, it also makes a comparison between injury rates among vaccinees and COVID-19 injury rates among infected individuals. 

I've now gone through and reviewed this paper and I'm sorry, but this study is not correct. That is, it contains two major method errors. First of all, the pool of people used as denominator when calculating the percentage of COVID-19 infected people who developed certain conditions due to the infection is greatly inadequate. Second, the vaccine injury rates among vaccinees are compared to disease injury rates among infected people when instead they should be compared to disease injury rates among the total pool of unvaccinated people. These method inadequacies have serious consequences. I'll explain what I mean.

Correct estimation of infection pool

When calculating the risk of developing a condition from an infectious disease, you need to make a correct assessment of how how large the pool of infected people is. And to do that, you need to make an estimate. Merely counting the number of people who've tested positive in a certain area isn't enough, as you need to include people who don't go test themselves because of being asymptomatic, or of not having the energy to do it due to their symptoms, or of lacking interest or knowledge about the infection et c. There may be many of different reasons. This means you need do make an estimate, otherwise the denominator in the calculation of the percentage of infected people who develops the condition becomes incorrect. 

I'll use the study Estimation of the Lethality for COVID-19 in Stockholm County published by the Swedish Public Health Agency as an example of a correctly calculated risk, based on an adequately defined denominator [2]. The fact that this is a calculation of the lethality percentage from COVID-19 and not the percentage of infection complications is irrelevant, the point is that the same mathematics used in this study should've been applied in the present CRI study. From the Swedish study, in translation:

"Recruitment was based on a stratified random sample of the population 0-85 years. In the survey we use, the survey for Stockholm County was supplemented with a self-sampling kit to measure ongoing SARS-CoV-2-infection by PCR test. The sampling took place from March 26 until April 2 and 18 of a total of 707 samples were positive. The proportion of the population in Stockholm County which would test positive was thus estimated at 2.5%, with 95% confidence range 1.4-4.2%."

For a complex reason, which I won't go into but is described in detail in the study text, one needs to use a slightly higher percentage when multiplying it with the total number of people in the pool, but that's of minor importance. Anyway, in this study they had to use the figure 3.1169% and when they multiplied it with the number of people in Stockholm County, 2 377 000, they got 74 089. This estimate was then the correct denominator to use when calculating the percentage of people who died from COVID-19 in Stockholm County during this time period. 

The numerator was the number of people who died in Stockholm County with a strong suspicion of COVID-19 as a cause, which was 432, no incorrectness there either, as long as a suspected cause number, not a diagnosed cause number, is also used as the numerator when calculating the lethality from the COVID-19 vaccine when the infection lethality and vaccine lethality rates are compared. 

So, what they found was that the lethality from COVID-19 in Stockholm County was 0.58%. This is a correct figure, as long as we keep in mind the fact that some of the suspected COVID-19 deaths may later become diagnosed as unrelated to the infection. The above is thus how the authors of the present study should've carried out their calculations but they didn't. From their text: 

"Each day in this SARS-CoV-2 analysis, persons with a new diagnosis of SARS-CoV-2 infection were matched to controls who were not previously infected. As in the vaccine safety analysis, persons could become infected with SARS-CoV-2 after they were already matched as controls on a previous day, in which case their data would be censored from the control group (along with their matched SARS-CoV-2–infected person) and they could then be included in the group of SARS-CoV-2–infected persons with a newly matched control. Follow-up of each matched pair started from the date of the positive PCR test result of the infected member and ended in an analogous manner to the main vaccination analysis, this time ending when the control member was infected or when either of the persons in the matched pair was vaccinated."

This excludes a considerable amount of infected persons in the total pool of 4.7 million people belonging to Clalit Health Services (CHS), the health care organization in question, who didn't go test themselves because of a number of reasons (being asymptomatic, not having the energy or interest for it, et c). 

If they'd used an adequate figure in the denominator, the percentage of people established to've developed certain conditions from COVID-19 would've gotten vastly lower. However, the percentage of people determined to've developed these conditions from the mRNA COVID-19 vaccines was fully correct since there are no unregistered vaccinated cases and therefore the registered figures are to be used. 

Now, there's a study titled The First Wave of COVID-19 in Israel - Initial Analysis of Publicly Available Data, published by Public Library of Science [3], which has estimated the percentage of infected people in Israel looking at partly the same time period as the CRI study. From the paper:

"Fig 7 shows the 7-day moving averages for TPR [Test Positivity Rate] and for the daily amount of COVID-19 tests performed in Israel between March 11 and April 25. Since the mean TPR reached its peak value of 10% around March 25, when the infection rate was also at its maximum, we can conclude that the true IR in Israeli population also did not exceed 10% at that time."

With an infection rate around 10%, the estimated number of infected of the 4.7 million people belonging to CHS would've amounted to 470 000 in March and April 2020. This number gives us a hint as for the size of the denominator which should've been used in the CRI study calculation instead of the figure of 233 392 confirmed diagnoses which was used. (No, not all confirmed infected and injured were included but I'll address that later.) In short, the pool of participants should've been added with a vast amount of both symptomatic and asymptomatic SARS-CoV-2 positive people who didn't develop these medic care necessitating conditions. 

How large then, exactly, should the denominator have been? Well, since the present study not only looked at conditions arising from people having the infection in March and April 2020 but looked at a much longer time period, from March 1, 2020 to to May 24, 2021, a number far greater than 470 000 should be applied. What we need is to estimate how many new infections arose among the 4.7 million patients belonging to CHS during these 15 months in question. For the calculation to be really accurate, we need the total, accumulated number of infections. And this number is found at the website Our World in Data, in the form of daily new estimated COVID-19 infections [4]. If we look at these figures, we find that this total, accumulated number lands around 2.4 million infected cases. That number also takes into account the fact that CHS encompasses 52% of Israel's population.

Further, in the CRI study's Figure 4, eleven adverse events after vaccination have been chosen for comparison with the occurence of these after infection, and we find the following excess risk numbers associated with COVID-19: Arrhythmia 0.166%, acute kidney injury 0.125%, pulmonary embolism 0.062%, deep-vein thrombosis 0.043%, myocardial infarction 0.025%, pericarditis 0.011%, myocarditis 0.011%, intracranial hemorrhage 0.008%, appendicitis 0.004% and lymphadenopathy 0.003%. As for herpes zoster infection, the study found that COVID-19 reduces instead of increases the risk of acquiring it, with 0.009%. 

Now, if we apply the laws of mathematics and re-calculate these numbers, taking into account that the pool of participants should've been 10 times larger, we get the following, more correct figures: Arrhythmia 0.0166%, acute kidney injury 0.0125%, pulmonary embolism 0.0062%, deep-vein thrombosis 0.0043%, myocardial infarction 0.0025%, pericarditis 0.0011%, myocarditis 0.0011%, intracranial hemorrhage 0.0008%, appendicitis 0.0004%, lymphadenopathy 0.0003% and herpes zoster infection -0.0009%. 

I'd here like to interpose the suggestion of reading through the English translation of the Swedish COVID-19 lethality study that I took up in the beginning of my text as a correct, comparative example [5]. This is the main paper that the Swedish equivalent to the Centers for Disease Control and Prevention, the Public Health Agency (Folkhälsomyndigheten), refers to when talking about the COVID-19 lethality here and it's put up on one of the major information pages of their website. I really recommend reading all of it, because it explains so well and in such detail how come this model of denominator calculation without exception must be used in studies like these, which aim to investigate the rate of injuries/complications arising from an infectious illness.

Correct estimation of exposure pool

Let's continue to the second method error of the CRI paper. The vaccine injury rates among vaccinees are in the study compared to disease injury rates among infected people, when instead they should be compared to disease injury rates among the total pool of unvaccinated people (the unvaccinated pool in the first part of the study is irrelevant as this merely constitutes a COVID-19 negative control group, incomparable to a real life pool of unvaccinated people). From the paper:

"To place the magnitude of the adverse effects of the vaccine in context, we also estimated the effects of SARS-CoV-2 infection on these same adverse events during the 42 days after diagnosis." 

Also, in the study's Figure 3 and Figure 4, injury rates among vaccinees and injury rates among infected people are directly compared. The problem is, this type of comparison simply cannot be done, i e, it's an incorrect comparison. This is because the alternative to getting a vaccination is to not get the vaccination, the alternative isn't getting the infection. Also, when comparing vaccinees to infected instead of unvaccinated people, the risk/benefit calculation derived from these figures becomes greatly inadequate. I'll explain what I mean.

Let me start by taking the potentially crippling condition myocarditis as an example, a COVID-19 vaccine injury which has been extra noted in media lately since it primarily affects very young adults and teenagers, among which the increased risk after vaccination is around 0.02% [6]. According to the study's data, there's a 0.003% increased risk of getting myocarditis after the vaccine, and since older individuals are included here, that's a correct figure. Further, according to the study's data, the increased risk of developing the condition after a confirmed COVID-19 infection is 0.011%. Since we in accordance with the laws of mathematics have corrected that figure though, it's now narrowed down to 0.0011%.

However, when comparing the risk of developing condition X from taking vaccine Y with the risk of developing condition X from not taking vaccine Y, you can't compare a pool of vaccinees with a pool of infected people. Because when you get the vaccine, there's a 100% risk of getting the "infection" (in this case with viral RNA), while in the case of not getting the vaccine, it doesn't imply a 100% risk of getting the infection (with the virus), but only a risk of somewhere between 3% and 15% [2, 7]. 

And as we have seen, in Israel during the analysis period of the present paper, the infection rate was around 10% [3]. This means that we have to multiply the figure 0.0011% by 0.1 to get the correct risk increase for people of acquiring myocarditis if they stay unvaccinated. And this in turn means that the risk increase for COVID-19 derived myocarditis for people who don't get the vaccine is as low as 0.00011%. Now we're suddenly in a whole different ballpark, as 0.003, the vaccine myocarditis risk increase figure, is 27 times as much as 0.00011. And this means, that as for myocarditis, the risk of acquiring it is 27 times higher if you get vaccinated as opposed to if you abstain.

Further, we have to apply this re-calculation to all the other re-calculated COVID-19 related injury data of the study as well, given that the unvaccinated don't have a 100% risk for infection but only 10%. What we then find, is that if you're unvaccinated, the correct COVID-19 derived risk increase figures for the eleven mentioned conditions are: Arrhythmia 0.00166%, acute kidney injury 0.00125%, pulmonary embolism 0.00062%, deep-vein thrombosis 0.00043%, myocardial infarction 0.00025%, pericarditis 0.00011%, myocarditis, as said, 0.00011% also, intracranial hemorrhage 0.00008%, appendicitis 0.00004%, lymphadenopathy 0.00003% and herpes zoster infection -0.00009%. 

On the other hand, for COVID-19 vaccinated people, the increased risks of developing a majority of these conditions are higher, and it's worth underlining that they're all serious afflictions (or, as for lymphadenopathy, can point to such). For myocardial infarction, the vaccine generated increased risk is 0.001%, for pericarditis, it's 0.001% as well, for myocarditis, it's, as mentioned, 0.003%, for appendicitis, it's 0.005%, for herpes zoster infection, it's 0.016% and for lymphadenopathy, it's as high as 0.078%.

Now as said earlier, not all confirmed infected belonging to CHS were eligible to participate in the study. The level of truly COVID-19 attributable complications should've been the same in the eligible group and the excluded group but we don't know how large this excluded group was. It may have been very small, in which case the above calculation is correct, or very large, or in between. But let's be generous and say the excluded group was as large as the eligible. According to the laws of mathematics, the above figures should then be doubled, and the COVID-19 derived risk increase figures for the eleven conditions then become: Arrhythmia 0.0033%, acute kidney injury 0.0025%, pulmonary embolism 0.0012%, deep-vein thrombosis 0.00086%, myocardial infarction 0.0005%, pericarditis 0.00022%, myocarditis 0.00022%, intracranial hemorrhage 0.00016%, appendicitis 0.00008%, lymphadenopathy 0.00006% and herpes zoster infection -0.00018%.

As you can see, even if we include a generous estimation of the excluded group, the risk increases of acquiring any COVID-19 derived form of a majority of these conditions are still larger for the vaccinees than for the unvaccinated individuals. One may here object and say that even if the vaccine increased the risk of developing these conditions, it actually also reduced the risk of developing a number of conditions. From the paper:

"The BNT162b2 vaccine appears to be protective against certain conditions such as anemia and intracranial hemorrhage. These same adverse events are also identified in this study as complications of SARS-CoV-2 infection, so it appears likely that the protective effect of the vaccine is mediated through its protection against undiagnosed SARS-CoV-2 infection, which may be undiagnosed either because of a lack of testing or because of false negative PCR results."

In the study's abstract, this is commented as well:

"In this study in a nationwide mass vaccination setting, the BNT162b2 vaccine was not associated with an elevated risk of most of the adverse events examined."

Well, that's something of a play with words, because if we look at Table 2 in the paper, where the entirety of the adverse events associated with the vaccine is listed, we find that in total, the risk reduction for serious conditions generated by the vaccine was 0.04%, while the total risk increase for serious conditions generated by the same was 0.14%; that is, a whole 3.5 times larger.

Interestingly, with their work including these errors, these authors have provided scientific validation of the growing suspicion that the COVID-19 vaccinated state gives rise to various serious injuries to a much greater extent than does the unvaccinated - which is the opposite of the message relayed in the study - because even if the figures used for comparison with the vaccine injury figures are inadequate, the other figures in the study are most likely not.

Towards the end of the paper, one of the problems discussed above is briefly mentioned:

"When a person decides to become vaccinated, this choice results in a probability of 100% for the vaccination, whereas the alternative of contracting SARS-CoV-2 infection is an event with uncertain probability that depends on the person, place, and time."

However, since omitting to include a calculation example with an adequate exposure pool, based on a correct infection pool and official infection rate figures has such a large impact on the main message of this study - changing it from defining the COVID-19 vaccinated state as less injurious than the unvaccinated, to the opposite - merely briefly mentioning it towards the end like this, as one among several limitations of the study, is so greatly misleading that it constitutes an error in itself.

Finally, and most importantly, there's a special reason why the method inadequacies discussed above have serious consequences in this particular case. That is, the Centers for Disease Control and Prevention (CDC), the major public health organization in the United States and an organization with profound influence on public health officials worldwide, refers to this study and its figures in their documents as a source to support their view that the benefits of COVID-19 vaccinating the population outweigh the risks of not vaccinating [8]. Of course, had the present study been correctly performed, it would've pointed the CDC in the direction of determining the opposite; that the risks of vaccinating are greater than abstaining.

References

1. Barda, N, Dagan, N, Ben-Shlomo, Y et al (2021) Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting N Engl J Med 385(12): 1078-1090 https://www.nejm.org/doi/10.1056/NEJMoa2110475

2. Svenska Folkhälsomyndigheten (2020) Skattning av Letaliteten för Covid-19 i Stockholms Län https://www.folkhalsomyndigheten.se/contentassets/da0321b738ee4f0686d758e069e18caa/skattning-letalitet-COVID-19-stockholms-lan.pdf

3. Last, M (2020) The First Wave of COVID-19 in Israel - Initial Analysis of Publicly Available Data PLoS One 15(10): e0240393 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7595440/ 

4. Our World in Data, Global Change Data Lab (2021) Daily New Estimated COVID-19 Infections from the ICL Model, Israel https://ourworldindata.org/search?q=israel+covid+estimated

5. The Swedish Public Health Agency (2020) Estimation of the Lethality for COVID-19 in Stockholm County Online translation of [2] via https://translate.google.com/?sl=sv&tl=en&op=docs

6. Vogel, G & Couzin-Frankel, J (2021) Israel reports link between rare cases of heart inflammation and COVID-19 vaccination in young men Science, American Association for the Advancement of Science https://www.science.org/news/2021/06/israel-reports-link-between-rare-cases-heart-inflammation-and-covid-19-vaccination

7. Angulo, F J, Finelli, L & Swerdlow, D L (2021) Estimation of US SARS-CoV-2 Infections, Symptomatic Infections, Hospitalizations and Deaths Using Seroprevalence Surveys (2021) JAMA Netw Open 4(1): e2033706 https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2774584 

8. Lee, G M & Hopkins, Jr, R H, Centers for Disease Control and Prevention, Advisory Committee for Immunization Practices [ACIP Workgroup Presentation] ACIP Meeting, Atlanta, GA, United States  (2021, August 30) COVID-19 Vaccine Safety Updates: COVID-19 Vaccine Safety Technical (VaST) Work Group Assessment https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2021-08-30/05-COVID-Lee-508.pdf