properly measuring vaccine efficacy and risk reward

a look at the UK vaccine surveillance data

cliff notes:

• measuring vaccine efficacy as % reduction in likelihood of severe outcomes can be misleading

• we must also measure absolute risk reduction. 50% drop from 20% risk is very different from 50% drop on 0.2% risk

• vaccines seem to show % efficacy in reducing hospitalization and death

• but for the young, healthy, and recovered, risk was already so low that the absolute drop does not look like good risk/reward vs side effect profile of the vaccines

• vaccines do not provide sterilizing protection against spread and seem to make it worse. there is no case to be made for societal obligation to vaccinate to protect others.

• mandating vaccination rather than allowing personal choice based on individual circumstance will inflict net harm on a great many people

• that’s immoral and represents medical malpractice

many claims are being made about covid vaccines and what they do and do not do. the problem is that nearly all the measurements here are being manipulated, misrepresented, and just plain old done wrong.

so let’s see if we can start with a clean sheet of paper and do it correctly because there is one simple, immutable fact here:

medical choices are everywhere and always a cost/benefit decision.

and thus we need a solid accounting of both.

the UK has, to my dismay, stopped including all the data in their variants of concern reports that i was using to track vaccine efficacy (VE) and case fatality rate (CFR) for delta variant. this has led me to shift to data from their vaccine surveillance reports. this will make it difficult to compare to prior numbers as it is not variant specific, but delta is so prevalent there that they should not be far off.

these reports run on a trailing 4 week basis so report 41 covers the 4 weeks from week 37 to week 40 of 2021 (9/13 to 10/10). i used this as the basis for this analysis. report HERE.

this report has, like most reports on this topic, a methodology designed to favor vaccines. it counts as “vaccinated” only those 14 days or more after their second dose of vaccine. this ignores that high risk period that comes after dose 1 and prior to actual efficacy a couple weeks post dose 2. i have discussed this issue in detail HERE.

there is a further issue in that the people who get one dose and then never progress to a second wind up showing VERY high risk, generally multiples of other classes including unvaccinated. this is frequently associated with some sort of bad reaction/poor tolerance. everyone who wind up getting double vaxxed takes this risk. no one knows, a priori, which bucket they will wind up in. excluding them or calling them “unvaxxed” is a nasty kind of bayesian manipulation. you have to include all the lottery players, not just the winners.

all this pops when you aggregate the cohorts in more sensible fashion. the UK uses these categories:

note that, oddly, there seems to be no accounting for the 14 days post dose 2. it seems not to be tracked or broken out and i’m really not sure where that data goes (if anywhere). it appears like a donut hole in the reporting that would seem to further benefit vaccines in terms of reported efficacy. i see no way to account of adjust for it, so i did the best i could without it.

i took all the categories of vaccinated that did exist and aggregated them. i then population normalized them to per 100k using the vaccinated percentages by age group in the same report.

we see data on cases similar to that discussed HERE, but with a couple notable variances:

1. relative to as reported (light blue) the big mover is the under 18’s, where few are vaccinated. honestly, the data there is such a mess, testing rates for school kids are so extreme, and intra-category vaccine variation (55% in 16-18, 0% under 12, etc) is such a data-salad of confounds and simpson’s paradox issues that i urge caution in comparing it to much of anything or placing much faith in it both above and in the data to come. it’s not terribly reliable.

2. after adjustment, cases are more common in the vaccinated than the unvaccinated in every age group except for 18-29. i suspect this one outlier is likely an artifact high testing rates imposed on the unvaxxed relative to the vaxxed, especially at colleges.

but also note the yellow lines. that’s the rate among those who got one dose but not a second. it’s clearly a highly enhanced risk category. this aligns with all cause deaths data by vaccine status which can be seen HERE.

but one thing is very clear from this data: the vaccines are not stopping spread.

the risk ratios make this clear.

some have argued that this is cohort bias with the healthy and prev recovered being less likely to vaccinate, but this seem unpersuasive in a place like UK where 95% of those over 65 are vaxxed and over 90% of over 50’s. i suspect it may well invert with only the most infirm/highest health risk for vaxx not being vaxxed.

so far, so (relatively) uncontroversial.

so now let’s look at what WILL be controversial, because i am now going to argue that basically everything you’ve been told about how to measure vaccine efficacy and thus risk reward is basically wrong, or, at best, materially misleading.

there are two ways to measure the efficacy of a vaccine.

the first is percentage benefit. you take the incidence per capita of an outcome for the unvaccinated and compare it to the vaccinated. of 10% of unvaxxed go to the ER and 5% of the vaxxed do, that’s 50% vaccine efficacy. simple enough. this is what has broadly been used.

BUT, this methodology has a significant limitation: it does not tell you how much good you really did.

• if i could reduce your lifetime risk of dying from cancer by 50%, that’s a big deal.

• if i can reduce your risk of being eaten by a shark by 50%, for all but a few folks, it’s basically meaningless.

this is to say that “magnitude matters.” public health officials and vaccine companies have not been addressing this. so let’s see if we can:

first, let’s look at percentage effects. i calculated risk of hospitalization and deaths (each for within 28 days of diagnosis, a highly inclusive definition that the UK reporting uses) for each age group to get hospitalizations per case (CHR) and fatalities per case (CFR) for the unvaccinated and for the aggregate of all those with one or more dose of vaccine. this raw figure is shown below.

(i suspect there may be meaningful testing variance favoring vaccines here based on protocols to test the unvaxxed by not the vaxxed on arrival at ER etc. i have no idea how to quantify or control for it and so have not included any attempts at dealing with it, but it is a limitation here that would be better addressed via RCT. )

in addition, i then risk adjusted this figure to account for the relative case rates in each group and cohort to account for the full risk faced.

risk per case X likelihood of getting a case = total risk

(if you have twice the risk of getting a case and half the risk of going to hospital, you wind up right where you started on hospitalization risk)

we see efficacy similar (in raw terms) to much reported data. but it drops quite a lot when adjusted for case prevalence to levels mostly in the 40’s. this aligns well with the other looks i took at this data (HERE) that led to ~50% drops in hospitalization which means we’re getting similar results by coming at it two different ways. this is encouraging.

it also raises an intriguing possibility that much of the apparent efficacy of vaccines likes in their driving false or non clinical positives for cases. if you swell the denominator with “non-case cases” it makes outcomes look less prevalent. there is nothing in this data to allow for confirmation or exclusion of this hypothesis. it seems fertile ground for investigation.

the same efficacy (and possible denominator salting issues) was evident case fatality rate (apart from under 18’s being wonky and as there were only 4 deaths total there out of 348,514 cases, frankly, i think that number is way past unreliable and should be ignored) it aligned pretty well with the ~60% VE i got for delta (see link above)

this issue sets up the next point though which is that while “reducing risk for 18-29 year olds by 80%” sounds impressive, it’s actually not that impactful.

you basically decreased the shark attack risk in illinois.

this becomes clear when we assess vaccine efficacy in a more traditional fashion:

what was the absolute reduction in overall risk?

note: these numbers likely run quite high in terms of actual societal risk as mild cases do not get reported and severe ones do. hospitalizations and deaths are better counted than infections. they almost certainly drop in absolute terms by half (or perhaps a lot more) when considering the overall population. but we do not have that data, so we work with what we do have.

suddenly, this looks VERY different. those under 50 saw a less than 1% drop in overall likelihood of hospitalization when you take the group as a whole. in under 18’s, it’s a drop of 4 per 10,000. (and again, experienced reality may be 50% or more lower)

dropping from 0.9% to 0.24% is a big % move, but tiny move in absolute terms.

deaths are similar with the young getting even lower absolute benefit (though the older are getting more)

full data here just for completeness. (note the simpson’s paradox in the aggregates where % VE looks negative for the group as a whole but not for the individual cohorts due to risk stratification by age and reported case prevalence in the young who are being tested feverishly by schools and thus over expressed in case counts)

from this, a quite interesting and useful set of calculations emerges.

the overall drop in absolute risk for a 40-49 year old from getting the vaccination is 1.14% on hospitalization and 0.12% on death. these are not big numbers and are, as discussed, almost certainly overstated. reality based on full infection (IHR/IFR instead of SHR/CFR) is probably 0.57% and 0.06% or less.

if i offered to sell you a magical insurance policy that would grant you this benefit, how much would you be willing to pay? not much i would wager.

(if you are willing to pay a lot for such, please do let me know. i’m sure i can fashion a derivative instrument to accommodate you)

now, admittedly, we’re getting into some meaningful error bars here and there is always the difficulty of accounting for testing levels and variance by age and the effects that might have, but i think this is giving us a pretty good sense of the ballpark on risk reduction.

but what of the risk from the vaccines themselves?

roughly 1.3% of younger patients taking pfizer needed medical care (these are cumulative because you need 2 doses) about 0.3% wound up in hospital. 35% were too sick to undertake daily activities and 22% were too sick to work. (interestingly, the reactions look worse in the young than the old)

so, already, we’re getting into a pretty rough pricetag for drops on the order of 0.5% on hospital risk and 0.05% on full infection death risk.

but we can stratify this further. risk of hospitalization for the healthy (with no comorbidities) drops over 90%. risk of death by 99%.

over 80% of those in hospital and 96% of those who died had two or more. (source)

easier to see like this and full walk though HERE.

this is MASSIVE risk reduction and lops 1-2 decimal places off the absolute risks shown above.

now the healthy unvaxxed 40-49 year old is looking at 0.27% risk of going to hospital and 0.003 of death on a confirmed case basis and likely half that on full infection basis. risks from the vaccine are higher for both and those are likely understated while the risks from covid likely overstated due to the 28 day follow period that’s catching “event with” instead of “event from.” those figures could easily be double actual event from as well.

for such people, there is essentially nothing to mitigate. now we’re not halving risk of shark attack in illinois (it has happened btw), we’re halving shark attack risk on the moon.

this same thing is true of those who have already had covid and recovered. immunity is long lasting, appears sterilizing, and is far more potent than vaccinated immunity. lots of data and studies in this excellent collection HERE.

reinfection rates for covid are vanishingly low, around 0.3% and 80% of those seem unserious. serious reinfection rate is ~0.06%. (DATA) and even those cluster in the infirm and elderly. (frankly, we’re getting so close to the error rate of the test it’s hard to even know if this is real)

all the data here is skewed to favor vaccines, but they are still not looking attractive for many, possibly most.

(recall that even the vaccine trials themselves failed to show all cause mortality benefit)

• if you are under 60 and healthy, the risk reward for these vaccines looks extremely poor.

• if you have had covid and recovered, the risk reward for these vaccines looks extremely poor.

• if you are both, the risk reward is outright absurd. there is no meaningful risk to mitigate in the first place.

reducing risk that low by half would not be worth crossing the street over, much less taking a vaccine with the highest rate of adverse events of any ever approved by the FDA by at least an order of magnitude, probably more.

if you are old, disease naïve, and have comorbidities, this may make a lot of sense, at least on a personal level.

the risk reward for you may be quite good, though there is a risk that leaky vaccines are making life worse for everyone by inverting the viral evolutionary gradient and selecting for hotter, more virulent strains instead of milder ones. (details HERE and HERE) if that is occurring (and it appears it may be though it’s only a hypothesis) then your vaccine is not protecting others, it’s threatening them, especially as it seems associated with higher case infection rates.

so there is really no case here for “vaccine as social obligation.” it does not stop spread, seems to make it worse, and may be making the virus worse as well.

this all comes down to personal risk reward. assuming it’s not a societal negative, those in high risk groups due to age, infirmity, and risk factor could benefit from being vaccinated.

the young, healthy, and recovered look to face deeply negative net expected outcomes from vaccines.

medicine is a PERSONAL decision and should be based on good data and individual circumstance.

in the face of data like this, mandates seem well beyond ill advised. they constitute medical malpractice.

get the data, check mine, evaluate your own circumstances, and make the choice that makes sense to you.

this sort of investment in understanding your personal well being pays life long dividends.