Mortality Combat: The War Over Counting the Dead in Iraq

By Sheldon Rampton

Mourners in Sadr City, August 2005 (AP Photo/Karim Kadim)

I was shocked when The Lancet, a highly respected British medical journal, published a study in October estimating that 655,000 Iraqis have died since 2003 as a result of the war. The number seemed huge, much larger than I had imagined possible.

The study, with Gilbert Burnham as its lead author, was conducted by some of the same researchers from Johns Hopkins University and Al Mustansiriya University in Baghdad who conducted a previous study in 2004 which estimated that 98,000 people had died. The earlier study was attacked at the time by supporters of the war and was largely ignored by the mainstream news media in the United States, as John Stauber and I noted in our recent book, The Best War Ever: Lies, Damned Lies and the Mess in Iraq (for an excerpt, see the Third Quarter 2006 issue of PR Watch). The new study suggests that some half a million additional lives have been lost in the subsequent two years.

Are these numbers credible? I looked at reactions to the Lancet study from several groups: American political pundits, scientists with expertise in health and mortality research, and Iraqis (as reflected in the views of Iraqis with English-language weblogs). Many of the political pundits (even those with anti-war views) either rejected the study or questioned its conclusions and methodology. The scientists, however, gave it high marks, and most of the Iraqis thought the number sounded like it was in the right ballpark.

What the Study Says

The full Lancet study is available online. Although it is a scientific paper, I found it easy to read and jargon-free. However, a couple of terms might need explanation.

The study uses a "cluster sampling" methodology that is commonly used in health and mortality research, especially in places hit by war or other humanitarian disasters such as floods or earthquakes. The methodology is somewhat less precise -- but more cost-effective and practical -- than simple random sampling, in which individual members of the population being studied are selected and interviewed at random. Rather than individuals, researchers interview randomly-selected clusters of individuals and use standard statistical techniques to reach conclusions about the entire population. As Daniel Engber explained in Slate magazine, "It's the same basic method used for political polls in America, which estimate the attitudes of millions of people by surveying 1,000 adults."

Currently the most comprehensive alternative attempt to compile statistics on Iraqi deaths is being done by the Iraq Body Count website, which as of November 2, 2006, had tallied 45,061 to 50,022 deaths -- less than a tenth of the Lancet result. As the Lancet paper itself notes, "Our estimate of excess deaths is far higher than those reported in Iraq . . . This discrepancy is not unexpected. Data from passive surveillance (counting the dead by external sources such as government or media) are rarely complete, even in stable circumstances, and are even less complete during conflict, when access is restricted and fatal events could be intentionally hidden. Aside from Bosnia, we can find no conflict situation where passive surveillance recorded more than 20% of the deaths measured by population-based methods."

Lancet editor Richard Horton explains the "active surveillance" methodology used in its study in a commentary published in the Guardian:

Only when you go out and knock on the doors of families, actively looking for deaths, do you begin to get close to the right number. This method is now tried and tested. It has been the basis for mortality estimates in war zones such as Darfur and the Congo. Interestingly, when we report figures from these countries politicians do not challenge them. They frown, nod their heads and agree that the situation is grave and intolerable. The international community must act, they say. When it comes to Iraq the story is different. Expect the current government to mobilise all its efforts to undermine the work done by this American and Iraqi team. Expect the government to criticise the Lancet for being too political. Expect the government to do all it can to dismiss this story and wash its hands of its responsibility to take these latest findings seriously.

Assessments from Scientists

Here are some of the reactions from scientists who work in the field of mortality research:

• Ronald Waldman, an epidemiologist at Columbia University who worked at the U.S. Centers for Disease Control and Prevention for many years, told the Washington Post that the Lancet's survey method was "tried and true" and said its findings were "the best estimate of mortality we have."

• According to Professor Frank E. Harrell Jr., chairman of the biostatistics department in the School of Medicine at Vanderbilt University, "The investigators used a solid study design and rigorous, well-justified analysis of the data. They used several analytic techniques having different levels of assumptions to ensure the robustness of mortality estimates and the estimated margin of error. The researchers are also world-class."

• Francisco Checchi, an epidemiologist at the London School of Hygiene and Tropical Medicine who has worked on mortality surveys in Angola, Darfur, Thailand and Uganda, said that he found the survey's estimates "shockingly high," but added that dismissing it "simply on gut feeling grounds seems more than irrational." He noted that its "choice of method is anything but controversial" and found its results "scientifically solid" and "compelling."

• In Australia, 27 of the country's leading scientists in epidemiology and public health signed a letter supporting the study, noting that it "was undertaken by respected researchers assisted by one of the world's foremost biostatisticians. Its methodology is sound and its conclusions should be taken seriously. ... The study by Burnham and his colleagues provides the best estimate of mortality to date in Iraq that we have, or indeed are ever likely to have."

Asked about the study at a news conference, President Bush dismissed it out of hand, calling it "not credible" and saying its methodology was "pretty well discredited."

"That's exactly wrong," responded Richard Garfield, a public health professor at Columbia University who works closely with a number of the authors of the report. "There is no discrediting of this methodology. I don't think there's anyone who's been involved in mortality research who thinks there's a better way to do it in unsecured areas. I have never heard of any argument in this field that says there's a better way to do it."

Politicians and Pundits

Most of the methodological criticisms of the Lancet study actually come from people like Bush who have no expertise in epidemiology, and of course the boldest attacks have come from supporters of the war.

Writing in the conservative National Review, Richard Nadler called the Lancet paper a "cooked up study." His only methodological critique, however, consisted of an odd claim that the researchers were guilty of "baseline bungling": they "chose their 'base-line' for pre-invasion Iraq as January 2002 through March 2003," a period Nadler argues was less violent than earlier periods of Saddam Hussein's rule.

Fred Kaplan in Slate magazine wrote that The Lancet's pre-war death estimate (5.5 Iraqis per 1,000) was flawed because it differed from an estimate of 10 per 1,000 published by the United Nations. Moreover, he says, a 5.5 per thousand prewar mortality rate would have been "lower than that of almost every country in the Middle East" (a claim made also by columnist William M. Arkin in the Washington Post). However, Australian computer scientist Tim Lambert has demolished the Kaplan-Arkin criticism in great detail, pointing out that in fact, 5.5 deaths per thousand is actually higher than the mortality rate in "all but one" of the other countries in the Middle East.

Another attempt at methodological criticism came from Republican pollster Steven E. Moore, who conducted surveys in Iraq and served as an advisor to Paul Bremer. Writing for the Wall Street Journal, Moore blasted the Lancet paper, calling it a "bogus study." His criticism focused on the study's allegedly too-small sample size and imprecision. "Survey results frequently have a margin of error of plus or minus 3% or 5% -- not 1200%," he wrote. This is generally true -- with regard to the sort of opinion surveys that Moore performs (although his research in Iraq left Bremer forced to admit belatedly that "we really didn't see the insurgency coming"). The Lancet study, however, was studying mortality, and its sample size was dictated in part by the limited funds available to finance it and in part by concern for the safety of the Iraqi researchers who conducted the survey.

Similar vitriol came from Christopher Hitchens, the former Trotskyist turned pro-war polemicist, who dashed off a column that didn't so much critique the Lancet paper as urinate on it. After accusing the epidemiologists of "moral idiocy," Hitchens mocked the name "Lancet," called its editor an "Islamist-Leftist," and went on to claim that its mortality estimate is "almost certainly inflated" and actually justifies the war. Why? The study found that 31 percent of deaths were attributed to coalition forces, while 24 percent were attributed to "other" causes and 45 percent were "unknown" (because either the responsible party was not known, or the surveyed households were hesitant to specifically identify them). From this evidence, Hitchens concluded that insurgents are the true killers in Iraq and that the Lancet study is therefore "a reminder of the nature of the enemy we face."

Another coffin in Baghdad, November 2006
(AP Photo/Karim Kadim)

Iraq Body Count

Other criticism of the study came from a source that may seem surprising: Iraq Body Count (IBC), the anti-war, London-based organization that has been tracking Iraqi deaths since the beginning of the war. IBC issued a news release questioning the wide gap that separates its own numbers and official Iraqi government statistics from the Lancet's much larger estimate. The discrepancy, they argued, is so large as to be implausible. For example, IBC doubts that the number of deaths estimated by the Lancet could have occurred "with less than a tenth of them being noticed by any public surveillance mechanisms." A gap that large, they argue, can only mean that either there has been "incompetence and/or fraud on a truly massive scale by Iraqi officials in hospitals and ministries," or else the Lancet authors "have drawn conclusions from unrepresentative data."

Les Roberts, one of the authors of the Lancet study, has responded to these criticisms in an interview with the British Broadcasting Corporation. Citing examples from other wars, he points out that "It is really difficult to collect death information in a war zone! . . . I do not think that very low reporting implies fraud."

It should be noted that IBC's own methodology follows rules that should be expected to lead to a lower count than the Lancet survey:

• Whereas the Lancet study attempts to estimate all deaths -- including the deaths of insurgents, police and Iraqi military -- IBC only counts civilian deaths and excludes combatants.

• IBC only counts deaths that are reported in English-language news media, and Iraq is not an English-speaking nation. Many more deaths are reported in the Iraqi press in Arabic than in the Western-language wire services.

As for the gap between the Lancet figure and deaths reported by the Iraqi Health Ministry, a number of Iraqi commentators (some of whom I quote below) have noted that conditions in many parts of the country as so unstable as to prevent reliable government accounting. Moreover, the question of how many people have died in Iraq has been politically charged since the start of the war, and the United States has not only avoided issuing statistics of its own but on a number of occasions has also pressured Iraqi officials against doing so. Shortly after the invasion in 2003, Baghdad's medical officials were forbidden to release morgue counts. In December of that year, Iraq's Health Ministry ordered a halt to counting civilian deaths and told its statistics department not to release figures, according to the Associated Press.

Iraqis Weigh In

Among Iraqi bloggers, the strongest challenge to the Lancet study came from Omar Fadhil, one of two brothers who contributes to a pro-occupation website called "Iraq the Model" (ITM). Fadhil emotionally blasted the study, accusing the Lancet researchers of

exploiting the suffering of people to make gains that are not the least related to easing the suffering of those people. . . . They shamelessly made an auction of our blood, and it didn't make a difference if the blood was shed by a bomb or a bullet or a heart attack because the bigger the count the more useful it becomes to attack this or that policy in a political race and the more useful it becomes in cheerleading for murderous tyrannical regimes.

These comments prompted an equally emotional outpouring from dozens of other Iraqi bloggers, who called ITM "a holocaust denier," "sucking up to the Americans," "a traitor," "like the Baathist apologist that they so despise," and "shameful." An Iraqi housewife declared that she was full of "Guilt and anger because the Iraq I always dreamt of has become one big nightmare. . . . Guilt and anger because outside these walls are trashbins filled with decapitated bodies of women, children and men. . . . Guilt and anger because after all the years of tyranny, people are now wishing for Saddam the criminal to come back. . . . The so called freedom that everyone, every single person was hoping and dreaming of has gone."

I spent some time sampling discussions of the Lancet study from among the more than 200 blogs listed at the Iraq Blog Count website. Many of the bloggers there noted that they themselves have seen widespread death due to the war, including the loss of personal friends and family: "I don't know of anyone who hasn't lost at least some members of their extended family," wrote Iraqi blogger Raed Jarrar.