More than 82,000 people in the United States have died of COVID-19 as of Tuesday. How many more lives will be lost? Scientists have built dozens of computational models to answer that question. But the profusion of forecasts poses a challenge: The models use such a wide range of methodologies, formats and time frames that it's hard to get even a ballpark sense of what the future has in store.
Enter Nicholas Reich, a biostatistician at University of Massachusetts Amherst. Reich and his colleagues have developed a method to compare and ultimately to merge the diverse models of the disease's progression into one "ensemble" projection. The resulting forecast is sobering. By June 6, it projects, the cumulative death toll in the U.S. will reach 110,000.
Reich's approach builds on work he has done over the past four years for the Centers for Disease Control and Prevention, pulling together the many forecasts that U.S. experts create annually to predict how that year's seasonal influenza will play out.
Reich's team has set up a similar system to compare coronavirus models. It's a sort of portal through which the scientists behind each COVID-19 model can communicate key details about their methodology and results so that, as Reich explains, "all of these forecasts can be represented in a single standardized way. And this makes it really easy to make apples-to-apples comparisons between these models."
The team unveiled the first version four weeks ago and ever since has been adding in more forecasts and updating the projections weekly. The latest update — released Tuesday — incorporates eight models, including some oft-cited ones, such as those built by the Imperial College London, the University of Washington Institute for Health Metrics and Evaluation, Columbia University and Northeastern University. (The team also sends each week's release to the CDC, which publishes a version with a slight time lag.)
The projections vary substantially — with the most pessimistic forecasting a total death toll of 120,000 by June 6 and the most optimistic forecasting 103,000 deaths by that date. But the models have been inching closer to each other. Over the past several weeks, the distance between the highest and lowest estimates has halved from a gap of 36,000 deaths two weeks ago to a gap of 17,000 deaths in the most recent update released Tuesday.
Still, says Reich, that remains a large difference. Also, he says, some of the models are gyrating fairly significantly from week to week.
"The most pessimistic model a few weeks ago was the model from Los Alamos National Laboratory," notes Reich. "Now Los Alamos is one of the most optimistic." Meanwhile, the models produced by IHME and University of Texas at Austin respectively have substantially increased their projected deaths tolls — becoming among the most pessimistic.
There are a range of reasons for these changes. The scientists are getting new data; they are updating their methods as they calibrate their models against the reality to date; and Americans have stopped social distancing to the same degree as they had been in March and April — requiring models that assumed a longer stay-at-home period to adjust their forecasts upward.
But how do we make sense of these COVID-19 projections if the models can see-saw so abruptly from week to week? That's where Reich's "ensemble" model may be helpful. It's a strategy that forecasters use regularly to model not just disease outbreaks, but other phenomena ranging from weather to electoral outcomes.
"Individual models are being changed every week. They're sensitive to the last observed data in different ways," says Reich. But with an ensemble "there's a certain consistency and robustness. You're not quite, sort of flapping in the wind."
Caitlin Rivers, an epidemiologist at Johns Hopkins Center for Health Security, agrees. "We know from the experience of seasonal influenza that ensemble models tend to perform better than any single model," she says.
And the approach is particularly well-suited in this pandemic given how much is still unknown. "This ensemble approach really pulls together those different results and makes them, in some ways, greater than the sum of their parts," says Rivers.
Reich thinks he can still improve his ensemble model. For instance, at the moment Reich is giving equal weight to each of the forecasts that go into it. But soon he hopes to give more weight to those that are proving more accurate — an approach that he uses in his ensemble models for flu.
"Model accuracy is one thing that we're tracking and we're hoping to release some information over the next few weeks," says Reich. "We've been sort of building the car as we're driving it at 90 miles an hour down the highway. And we're learning as we go."
In the meantime, says Rivers, it is at least useful to be able to see the range of forecasts that have been produced. "It tends to help the user to understand where the uncertainties are and the spectrum of possible futures," she says.
That said, Reich's setup compares projections through the next four weeks only — since many of the models don't offer forecasts beyond that point. This is why the latest update his team released projects deaths up to June 6.
Still, in that first week of June about 7,000 people are projected to die. The clear implication, adds Reich: The cumulative death toll of 110,000 forecast through June 6 will unquestionably rise much higher in the weeks beyond.
STEVE INSKEEP, HOST:
The coronavirus has now killed more than 82,000 people in the United States. How many more will die? Scientists have built different models that answer that vital question differently. NPR's Nurith Aizenman reports on one team that has tried to make sense of competing projections.
NURITH AIZENMAN, BYLINE: Americans got a glimpse of just how many different forecasts there are for this pandemic at a White House press conference last March. Dr. Deborah Birx of the Coronavirus Task Force started thanking all of the modelers she'd been consulting.
(SOUNDBITE OF PRESS CONFERENCE)
DEBORAH BIRX: There was a modeler out of the University of Washington, from Harvard, from Columbia, from Northeastern, from Imperial.
AIZENMAN: Since then, the number of epidemiologists and statisticians putting together COVID-19 projections has only multiplied. We're talking dozens. And that makes it challenging to get even a ballpark idea of where the pandemic is headed.
NICHOLAS REICH: Each of these models has their own perspective.
AIZENMAN: Nicholas Reich is a biostatistician at University of Massachusetts Amherst.
REICH: They're using different data sources, using different methodologies.
AIZENMAN: It's not even possible to make side-by-side comparisons. Each model uses slightly different formats, time frames. So Reich and his colleagues at Amherst have come up with a kind of portal through which each modeler can communicate their findings. Now...
REICH: All of these forecasts can be represented in a single, standardized way.
AIZENMAN: The most pessimistic forecast - by June 6, 120,000 people will have died. The most optimistic - 103,000 dead by that date, which is as far out as Reich's roundup goes. He's also produced what's called an ensemble model, a merging of all the forecasts into one single projection. In this case, by June 6, the death toll will reach 110,000. Caitlin Rivers is an epidemiologist at Johns Hopkins University.
CAITLIN RIVERS: Having this ensemble approach really pulls together those different results and makes them, in some ways, greater than the sum of their parts.
AIZENMAN: She notes that when it comes to forecasting yearly outbreaks of the flu, ensemble models tend to outperform any single model. In fact, Reich's expertise comes from years of working with the U.S. Centers for Disease Control and Prevention to create ensemble models of seasonal flu. The ensemble approach also helps smooth out the weekly changes among many models as their creators have tweaked and updated them. James Scott leads a team that created a forecast by the University of Texas at Austin.
JAMES SCOTT: If you've seen our model getting more pessimistic over the last, say, seven to 10 days, it's because there's been a notable uptick in mobility patterns around the United States.
AIZENMAN: More people interacting face-to-face means more infections and more death. But other models have become more optimistic. Meanwhile, Reich thinks he can still improve on his ensemble model. Right now, it's effectively an average of all the projections that go into it. Eventually, he hopes to give more weight to the projections that have proven more accurate.
REICH: We've been sort of building the car as we're driving it at 90 miles an hour down the highway and we're learning as we go.
AIZENMAN: To see how the models line up and how they've changed over time, you can go to npr.org.
Nurith Aizenman, NPR News. Transcript provided by NPR, Copyright NPR.