• Email
  • Single Page
  • Print

Health Care: Who Knows ‘Best’?

Cass Sunstein; drawing by John Springs

What may account for the repeated failures of expert panels to identify and validate “best practices”? In large part, the panels made a conceptual error. They did not distinguish between medical practices that can be standardized and not significantly altered by the condition of the individual patient, and those that must be adapted to a particular person. For instance, inserting an intravenous catheter into a blood vessel involves essentially the same set of procedures for everyone in order to assure that the catheter does not cause infection. Here is an example of how studies of comparative effectiveness can readily prove the value of an approach by which “one size fits all.” Moreover, there is no violation of autonomy in adopting “aggressive” measures of this kind to assure patient safety.

But once we depart from such mechanical procedures and impose a single “best practice” on a complex malady, our treatment is too often inadequate. Ironically, the failure of experts to recognize when they overreach can be explained by insights from behavioral economics. I know, because I contributed to a misconceived “best practice.”

My early research involved so-called growth factors: proteins that stimulate the bone marrow to produce blood cells. I participated in the development of erythropoietin, the red cell growth factor, as a treatment for anemic cancer patients. Erythropoietin appeared to reduce the anemia, lessening the frequency of transfusion. With other experts, I performed a “meta-analysis,” i.e., a study bringing together data from multiple clinical trials. We concluded that erythropoietin significantly improved the health of cancer patients and we recommended it to them as their default option. But our analysis and guidelines were wrong. The benefits ultimately were shown to be minor and the risks of treatment sometimes severe, including stroke and heart attack.4

After this failure, I came to realize that I had suffered from a “Pygmalion complex.” I had fallen in love with my own work and analytical skills. In behavioral economics, this is called “overconfidence bias,” by which we overestimate our ability to analyze information, make accurate estimates, and project outcomes. Experts become intoxicated with their past success and fail to be sufficiently self-critical.

A second flaw in formulating “best practices” is also explained by behavioral economics—“confirmation bias.” This is the tendency to discount contradictory data, staying wed to assumptions despite conflicting evidence. Inconsistent findings are rationalized as being “outliers.” There were, indeed, other experts who questioned our anemia analysis, arguing that we had hastily come to a conclusion, neglecting findings that conflicted with our position. Those skeptics were right.5

Yet a third powerful bias identified in behavioral economics can plague expert panels: this is the “focusing illusion,” which occurs when, basing our predictions on a single change in the status quo, we mistakenly forecast dramatic effects on an overall condition. “If only I moved from the Midwest to sunny California, I would be so much happier” is a classical statement of a focusing illusion, proven to be such by studies of people who have actually moved across the country. Another such illusion was the prescription of estrogen as the single remedy to restore feminine youth and prevent heart disease, dementia, and other complications of the complex biology of aging.6 Such claims turned out to be seriously flawed.

There is a growing awareness among researchers, including advocates of quality measures, that past efforts to standardize and broadly mandate “best practices” were scientifically misconceived. Dr. Carolyn Clancy of the Agency for Healthcare Research and Quality, the federal body that establishes quality measures, acknowledged that clinical trials yield averages that often do not reflect the “real world” of individual patients, particularly those with multiple medical conditions. Nor do current findings on best practices take into account changes in an illness as it evolves over time. Tight control of blood sugar may help some diabetics, but not others. Such control may be prudent at one stage of the malady and not at a later stage. For years, the standards for treatment of the disease were blind to this clinical reality.7

Orszag’s mandates not only ignore such conceptual concerns but also raise ethical dilemmas. Should physicians and hospitals receive refunds after they have suffered financial penalties for deviating from mistaken quality measures? Should public apologies be made for incorrect reports from government sources informing the public that certain doctors or hospitals were not providing “quality care” when they actually were? Should a physician who is skeptical about a mandated “best practice” inform the patient of his opinion? To aggressively implement a presumed but still unproven “best practice” is essentially a clinical experiment. Should the patient sign an informed consent document before he receives the treatment? Should every patient who is treated by a questionable “best practice” be told that there are credible experts who disagree with the guideline?

But even when there are no coercive measures, revising or reversing the default option requires a more complicated procedure than the one described by the President at the White House meeting. In November, the United States Preventive Services Task Force, reversing a long-standing guideline, recommended that women between the ages of forty and forty-nine do not need to have routine mammograms. To arrive at this conclusion, researchers made both a meta-analysis and computer models of data from seven clinical trials. The task force found that routine mammograms result in a 15 percent reduction in the relative risk of death from breast cancer for women in the forty to forty-nine age group, a similar level of benefit as in earlier analyses. For women in their forties, this means one life is saved for every 1,904 women screened. For older women in their fifties, one life is saved for every 1,359 women screened.8

If these estimates are correct, then how many lives might be saved in the United States for each age group if every woman received a mammogram? The 2008 US Census estimates the number of women between forty and forty-nine at 22.3 million. So if mammography were available to all these women, nearly 12,000 deaths could be potentially averted during these ten years in their lives. As for the 20.5 million women in their fifties, some 15,000 deaths could potentially be averted.

What are the risks of mammography for women in their forties? The task force estimated a higher rate of false positive findings in mammograms in women in their forties compared to older women. This translates into increased anxiety when women are told that there may be a cancer and there is not. A false positive reading may also result in a woman having a biopsy. For every case of invasive breast cancer in a young woman diagnosed by mammography, five women with benign findings will have biopsies. In addition, there are potential risks of radiation from the mammogram itself, although no one really knows how significant these are. Then there is an unanswered question in the biology of breast cancer: Which tumors are indolent and which are aggressive? We lack the molecular tools to distinguish between slow- and fast-growing cancers. Some slow-growing ones detected in young women might be treated later in life without any disadvantage in the rate of survival. But aggressive breast cancers in young women are notoriously difficult to treat and frequently result in death. And as with essentially all screening tests in a population, the majority of women receiving mammograms do not have any disorder.

These, roughly, are the statistics and state of the science with regard to breast cancer. How do we weigh the evidence and apply it to individuals and to society at large? Setting the default option that doctors will present to patients requires us to make value judgments. Dr. Otis Brawley of the American Cancer Society, an oncologist who worked for decades at the National Cancer Institute, is well versed in preventive care; he disagrees with the new default setting, based on findings that mammograms save lives. (Brawley also happens to be an African-American and has long been concerned about the meager access among minority and poor groups to potentially lifesaving screenings.)

Dr. Diana Petitti, a professor of bioinformatics at Arizona State University and vice-chair of the task force, appeared with Brawley on November 17, 2009, on the PBS NewsHour. She had no disagreement with him about what the studies show, and emphasized that the task force did not say that women in their forties should not get mammograms, only that they were no longer routinely recommended since the benefit to patients did not clearly outweigh the risks. Cost considerations were not part of the task force’s deliberations.

Other supporters of the new recommendations took a less temperate view. A statistician who developed computer models for the task force told The New York Times that “this decision is a no-brainer.”9 It did not appear to be so clear to Melissa Block of NPR when she interviewed an internist who agreed with the task force. The doctor said that stopping routine mammography for young women would spare them anxiety, distress, and unnecessary biopsies. Block replied, “I’ve heard this before…. When people say, you know, there’s unnecessary anxiety and false positives and fear and worry.” That, she said, is “a very patronizing approach to take toward women’s health…. Women may very well be willing to assume those harms if it means that they may be diagnosed earlier.” The internist replied that each woman should talk with her doctor and figure out what is best.10 Sunstein’s Nudge coauthor, the behavioral economist Richard Thaler, wrote a thoughtful analysis of the pros and cons of mammography in The New York Times and concluded that “one can make a good case that we don’t want the government making these choices” for us.11

Two days after the task force recommendations were released, Health and Human Services Secretary Kathleen Sebelius put some distance between the Obama administration and the task force’s conclusions, saying:

My message to women is simple. Mammograms have always been an important life-saving tool in the fight against breast cancer and they still are today. Keep doing what you have been doing for years….

Dr. Petitti later appeared before Congress to apologize for any “confusion” caused by the task force report. Petitti was not recanting a scientific truth. She correctly described the new recommendations as “qualitative.” That is, they were offered as value judgments that could be modified or revised; and the political process offers one way of doing so. As Sunstein has written, if default options embody standards that many people judge as not better for themselves, those standards can be changed.

Shortly after the new mammography guidelines were announced, an expert panel of obstetricians and gynecologists recommended that teenage girls no longer have routine pap smears for cervical cancer.12 The incidence of deadly cervical cancer among teens is at most one in a million and screening does not appear to save that one life. When false positive results from screenings are followed by cervical surgery, the risk may be injury that can predispose a young woman to later premature labor. There was no public uproar following this changed default setting for many women. It was consistent with how most people value the benefit of lives saved versus risks incurred. This is the reality of “comparative effectiveness” research. It is not simply a matter of “what works and what doesn’t.” Nor will patients always “insist” on being treated according to what experts define as “best practice.” They should be aware that there are numerous companies, some of them “not for profit,” issuing standards for treatment that are congenial to the insurance industry but are often open to the kinds of counterevidence I have described here.

What of the President’s statement that doctors will want to engage in federally approved “best practices”? The American College of Physicians, composed of internists, agreed with the task force conclusions about mammography. The American Society of Clinical Oncology, representing oncologists, did not. I am a member of both professional organizations. What do I do? As a physician who has cared for numerous young women with breast cancer, many dying an untimely death, my bias was that the dangers of mammograms do not outweigh the reduction in mortality. Notably, the oncologists who head the breast cancer programs at Minnesota’s Mayo Clinic and Utah’s Intermountain Health—described by President Obama as pinnacles of quality care using guidelines—also disagreed with the task force.

Such challenges to “best practice” do not imply that doctors should stand alone against received opinion. Most physicians seek data and views on treatments from peers and, as needed, specialists, and then present information and opinion to patients who ultimately decide.

While costs were not part of the task force calculations, they prominently entered the national debate on them. Dr. Robert Truog of Boston Children’s Hospital allowed that mammography saves lives, but asked if it is “cost effective.”13 That is, should policy planners set a price on saving those young women?

Cost-effectiveness is going to be a hard sell to the American public, not only because of the great value placed on each life in the Judeo-Christian tradition, but because the federal government has devoted many hundreds of billions of dollars to bail out Wall Street. To perform mammograms for all American women in their forties costs some $3 billion a year, a pittance compared to the money put into the bank rescue. The Wall Street debacle also made many Americans suspicious of “quants,” the math whizzes who developed computer models that in theory accurately assessed value in complex monetary instruments but in fact nearly brought down the worldwide financial system. When a medical statistician says that imposing a limit on mammography is a “no-brainer,” people may recall George Tenet’s claim that the case for invading Iraq was a “slam-dunk.”

At the White House gathering, the President portrayed comparative effectiveness as equivalent to cost- effectiveness, noting that other countries spend half of what we do by only paying for “what works.” This contention is not supported by evidence. Theodore Marmor, a professor of health care policy at Yale, writes in Fads, Fallacies and Foolishness in Medical Care Management and Policy that movements for “quality improvement” in Britain have failed to reduce expenditures.14 Marmor, with Jonathan Oberlander, a professor at the University of North Carolina, has written in these pages that the President has offered up rosy scenarios to avoid the harsh truth that there is no “painless cost control.”15 Lower spending in countries like France and Germany is accounted for not by comparative effectiveness studies but by lower costs of treatment attained through their systems of medical care and by reduced medical budgets. In Europe, prescription drugs cost between 50 and 60 percent of what they do in the US, and doctor’s salaries are lower. (Insurance premiums also are tightly constrained.) France and Germany have good records in health care, but in Great Britain, where costs are strictly controlled by the National Health Service, with rationing of expensive treatments, outcomes for many cancers are among the worst in Europe.16

The care of patients is complex, and choices about treatments involve difficult tradeoffs. That the uncertainties can be erased by mandates from experts is a misconceived panacea, a “focusing illusion.” If a bill passes, Cass Sunstein will be central in drawing up the regulations that carry out its principles. Let’s hope his thinking prevails.

—January 14, 2010


Health Care: Who Knows Best?’ April 8, 2010

Big Pharma & ‘Best’ Medical Practices March 11, 2010

  1. 4

    The clinical development of other growth factors, like G-CSF for a low white blood cell count, fared better. G-CSF is a valuable treatment for many cancer patients, but, of course, not all.

  2. 5

    Contradictory evidence reverses “best practices” so frequently that within one year 15 percent must be changed, within two years, 23 percent are reversed, and at 5.5 years, half are incorrect. See Kaveh G. Shojania et al., “How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis,” Annals of Internal Medicine, August 21, 2007.

  3. 6

    Focusing illusions are wonderfully illuminated by Daniel Gilbert, Stumbling on Happiness (Knopf, 2006). Also see the role of marketing in fostering the illusion: Natasha Singer and Duff Wilson, “Menopause, as Brought to You by Big Pharma,” The New York Times, December 13, 2009. See also David A. Schkade and Daniel Kahneman, “Does Living in California Make People Happy? A Focusing Illusion in Judgments of Life Satisfaction,” Psychological Science, September 1998.

  4. 7

    Dr. Clancy seeks new statistical methods to analyze heterogeneous groups of “real world” patients, so treatment guidelines become “personalized,” delivering “the right treatment to the right patient at the right time.” (See Patrick H. Conway and Carolyn Clancy, “Comparative-Effectiveness Research —Implications of the Federal Coordinating Council’s Report,” The New England Journal of Medicine, July 23, 2009; Harold C. Sox and Sheldon Greenfield, “Comparative Effectiveness Research: A Report From the Institute of Medicine,” Annals of Internal Medicine, August 4, 2009.) This is a laudable goal and deeply attractive. It is more likely to come from basic science that classifies patients based on their genetic characteristics rather than statistics. Past attempts at observing groups of “real world” patients have often generated conclusions that were flawed, mistaking correlation for causation. A valiant attempt to apply research on comparative effectiveness to prostate cancer treatment options came up against similar hurdles. See Jenny Marder, “A User’s Guide to Cancer Treatment,” Science, November 27, 2009.

  5. 8

    US Preventive Services Task Force, “Screening for Breast Cancer: US Preventive Services Task Force Recommendation Statement,” Annals of Internal Medicine, November 17, 2009; Heidi D. Nelson et al., “Screening for Breast Cancer: An Update for the US Preventive Services Task Force,” Annals of Internal Medicine, November 17, 2009; Jeanne S. Mandelblatt et al. for the Breast Cancer Working Group of the Cancer Intervention and Surveillance Modeling Network (CISNET), “Effects of Mammography Screening Under Different Screening Schedules: Model Estimates of Potential Benefits and Harms, Annals of Internal Medicine, November 17, 2009.

  6. 9

    Gina Kolata, “In Reversal, Panel Urges Mammograms at 50, not 40,” The New York Times, November 17, 2009. A detailed summation of the controversy is found in The Cancer Letter, November 20 and December 4, 2009.

  7. 10

    National Public Radio, All Things Considered, November 16, 2009.

  8. 11

    Richard H. Thaler, “Gauging the Odds (and the Costs) in Health Screening,” The New York Times, December 20, 2009.

  9. 12

    acog Practice Bulletin, “Clinical Management Guidelines for Obstetrician-Gynecologists,” Number 109, Obstetrics & Gynecology, December 2009; Denise Grady, “Guidelines Push Back Age for Cervical Cancer Tests,” The New York Times, November 20, 2009.

  10. 13

    Robert D. Truog, “Screening Mammography and the ‘R’ Word,” The New England Journal of Medicine, December 24, 2009.

  11. 14

    World Scientific, 2007.

  12. 15

    Health Reform: The Fateful Moment,” The New York Review, August 13, 2009.

  13. 16

    Theodore Marmor, Jonathan Oberlander, and Joseph White, “The Obama Administration’s Options for Health Care Cost Control: Hope versus Reality,” Annals of Internal Medicine, April 7, 2009; Donald M. Berwick, “Measuring Physicians’ Quality and Performance, The Journal of the American Medical Association, December 9, 2009. A layman’s journey seeking care abroad is described in the lively book by T.R. Reid. The Healing of America: A Global Quest for Better, Cheaper, and Fairer Health Care (Penguin, 2009). Concerning the uproar in Britain about poor cancer outcomes, see Nick Triggle, “NHS Must Get Better at Early Cancer Diagnosis,” BBC News, November 30, 2009; Rebecca Smith, “Cancer Care on the NHS Falls Behind the Rest of Europe,” The Daily Telegraph, November 30, 2009; NHS Department of Health, “Cancer Reform Strategy: Achieving Local Implementation—Second Annual Report,” December 1, 2009.

  • Email
  • Single Page
  • Print