Robert Wack

(originally posted on Your Angry Pediatrician on Substack, lightly edited)

This is a story of how in science, despite our attempts to achieve some understanding of objective reality, context still matters. What seems like a good idea in one instance can become a really bad idea in another, not because the science changed, because it can, but more often because the humans changed – society, politics, human behavior. This issue is the kind of thing, in the wrong hands, can be used to create disinformation. The larger issues are about the importance of process, credibility, and transparency in our health system to maintain trust.

Back in 1998, I wrote this letter weighing in on the debate at that time about the wisdom of the new (at that time) ACIP recommendation for universal varicella immunization for American children. My arguments were mostly directed at the ACIP’s heavy reliance on this study that used detailed economic modeling to conclude it made sense to recommend universal varicella vaccination due to the realization of “societal savings”.

The lead author on the economic benefits study is a pediatrician named Dr. Tracy Lieu. I didn’t know it at the time, but she and I are exactly the same age. She published that paper in 1994, a year after finishing a Masters in Public Health at Berkeley. Given the long lead times for medical publishing, it’s likely the work on the model was something she did as part of her Master’s program. She finished her peds residency in 1991, did the MPH, and was a freshly minted pediatrician, like me, in 1994.

When I was researching my letter back in 1998, I talked to a lot of people, including Dr. Lieu. This was pre- a lot of things, so I didn’t know as much about her then as I just learned with a few keystrokes writing this now. I remember our conversation back then as being cordial and professional, but clearly I irritated her with my persistent questions about the assumptions used to build her model, and the weights assigned to variables and relationships. Had I known we were peers back then, I probably would have pushed harder. 

If you have the time to dive into the paper, you’ll see the cost/benefit analysis revolves around several groups of items: on the vaccination side, it’s the costs of the vaccine plus administration; on the status quo side, it’s the medical costs of varicella illness, as well as the “societal costs” of each illness. This is what set my spider senses tingling back then. “Societal costs”?

You see this in public health discussions a lot, trying to estimate the social benefits and costs of an intervention. Take clean water for example. It costs a tidy sum to treat and distribute water to everyone in a community so they have clean tap water. That’s a straight up business calculation, understanding and quantifying the costs of an industrial process. What’s the cost if you don’t do it? Drinking impure water, a lot of people may get incapacitating diarrhea, and some of them may die. People sitting on the toilet can’t work, some of them may have to seek treatment. The lost wages, lost productivity, and cost of the hospital visits for the ill people all have discrete, pretty well known dollar costs to them. Add in lost economic development potential of a community always running to the bathroom, the investments not made, the businesses that close and move away, and you can derive a number to put up against cost of building your water treatment plant to see that, clearly, it makes sense to spend the money for clean water just based on economics, never mind the unpleasantness of constant diarrhea.

Dr. Lieu’s paper did a similar analysis for varicella vaccination. She looked at the cost of an illness, using assumptions about how often kids and adults were hospitalized with chicken pox before vaccines, how many died, what that treatment costs are, etc. The medical costs, though widely variable across the country, could still be reasonably estimated. So far, so good.

The “societal costs” are where things got a little sketchy. Those costs were attributed to missed school, additional child care expenses, missed work by parents, and over the counter medications like benadryl and calamine for itching. All reasonable on the surface, but the devil is in the details. What assumptions were made about the hourly cost of child care? How many kids need extra childcare as opposed to a parent or sibling caring for them? What if a parent isn’t working, so there’s no missed work expense? The assumptions around this seemed to me to be more than a little arbitrary.

Why does that matter? The thing about models is, no matter how careful you are building them, the more complex they are, the more they can produce unexpected outcomes based on what assumptions you feed into them. As you refine the model, tweaking values and relationships between variables, you can gradually bring it closer and closer to reality, but there’s always the risk you are missing something, a value of a variable, or a relationship between variables, that can dramatically influence the outputs in unanticipated ways. Statisticians and epidemiologists guard against this with a variety of techniques, like sensitivity analysis, where you tweak a variable to see how much influence it has to be sure you haven’t misweighted something. Still, it’s tricky work.

But there’s another risk: steering the model to the outcome you prefer. That steering doesn’t have to be intentional, but it can happen nonetheless. This is the accusation climate change deniers make about the global climate models. The climatology community gets around this by having multiple independent models and looking at the aggregate outputs across several models. When you have multiple models agree independently, you can be more confident in the reliability. But that’s not what we had back in 1998 with varicella. We just had this one model.

One of the most telling details was the assumption on the cost side that only one dose of the vaccine would be necessary, which significantly constrained the cost of the intervention, making it easier to show a benefit from the “savings”. The problem is, there aren’t any single dose vaccinations out there – one dose and you are done for life. You either need multiple doses for effective immunity, or periodic boosters to maintain immunity. Was it really reasonable to assume one dose per child for life? More than one dose in the model, and now the “savings” from universal vaccination disappeared.

What about deaths? Pre-vaccination, the widely cited statistic for varicella deaths in the U.S. was 100 per year, the majority of those being adults. Most of the pediatric deaths were in kids with underlying immune disorders, or cancer patients undergoing chemotherapy. Tragic no doubt, but relative to the total population of the U.S., a very small number.

This is what first made wonder why were doing this. If those death numbers were so small, why not focus immunization on vulnerable children and non-immune adults who escaped childhood without the rite of passage of a week off school and scratching the scabs you weren’t supposed to touch? Why spend the money on universal vaccination? This is where the human element enters the situation.

Anne Gershon is another pediatrician, a renowned infectious disease researcher who is a pioneer in the field. She was heavily involved from the beginning in the research behind the development of the varicella vaccine, work that was funded in part by Merck, the main manufacturer back then, of the varicella vaccine. Dr. Gershon was also a member of the ACIP, the body deliberating about and ultimately recommending the universal vaccination recommendation. 

Dr. Gershon saw the worst of the worst of varicella infections, as a peds ID consultant at a major academic medical center, where the immunosuppressed kids with raging chicken pox cases would end up. Some of those children died despite the best efforts of the team caring for them, something I’m sure Dr. Gershon witnessed. I completely understand why she would have strong feelings about the need for a varicella vaccine.

But when you are making an argument for a public health intervention based on “societal costs”, the feelings of the advocates, no matter how passionate and well-justified, shouldn’t enter the discussion. It’s like a bar fight: if you pick up a bottle, you better be ready for the other guy to pick one up too, and be ready to finish the fight. If you make a vaccine policy proposal based on dollars and cents, then you should be prepared for the decision to be challenged based on dollars and cents.

Back in 1998, I had the opportunity to discuss this with another pediatrician I had access to who was also a member of the ACIP committee at the time, and participated in the discussions about whether to recommend universal vaccination. As you often see with committees, in politics and healthcare, the unanimity of the final decision obscures the often strong disagreements between the various points of view that ended up losing the vote. After the vote, everyone shuts up and gets behind the conclusion.

I asked about what role Dr. Gershon played in the discussion. “She was relentless.” Apparently, Dr. Gershon was the driving force behind the ultimate recommendation for universal recommendation. There were certainly others as well, and a consensus once achieved is still a consensus, but as often happens, it takes a leader, a strong personality to forge a consensus out of a group of bright, opinionated experts sitting around the big table.

It was gratifying to hear, and also see in the literature, I was not alone in voicing my concerns about the process and assumptions about the decision to recommend universal vaccination. But after that conversation, and seeing the follow ups to my letter, both agreeing and attacking my argument, I was content to let it go, knowing that the argument had been made but didn’t prevail. Seeing a few years later the decision that, whoopsie, just kidding, now kids need TWO doses of varicella vaccine, was also gratifying.

Today, many fewer kids suffer through chicken pox, vulnerable adults now have an option for protection for their more serious infections, the immunosuppressed kids are protected, all of which are good things. Whether the original cost/benefit analysis still holds up is moot. We decided, it’s been implemented, the vaccine is safe and effective, so let’s move on.

And, now we have this new information about shingles and the possible protective effects of the vaccine against dementia, something that wasn’t even a flicker of dream back in 1998. Sure, there was discussion about shingles, but it was more about shingles forcing the need for MORE shots, undermining the economic benefit argument, but that wasn’t well quantified at the time and wasn’t a big part of the discussion, at least not for me. The economic upside of a 20% decrease in dementia, though difficult to quantify, is surely huge.

Add in the anti-vax craziness that now rages, and the ability to have vigorous, good faith arguments about the pros and cons of a particular public health intervention, whether about varicella, covid, or anything else, is much, much more difficult. Despite my concerns, or rather, because my concerns were honestly and thoroughly vetted and debated by people far smarter and more experienced than me, I trusted the system and sat down and shut up. Today? You wouldn’t hear a peep out of me today, because some bad faith troll would twist my words into some new anti-vax disinformation crusade.

Context matters. Trust matters. People can change their minds with new information. That’s how good science and public health policy works. But with all the bad faith grifters, phonies, and kooks out there today, it is infinitely harder to do that important, lifesaving work.