Will Hospital Report Cards Make the Grade?

Issue Brief No. 11
July 1997

ospital report cards that document patients’ medical outcomes are attracting increasing attention for their role in guiding health care decisions by employers, consumers and providers. Significant questions remain, however, regarding the validity and utility of this information. This Issue Brief is based on a seminar held by the Center for Studying Health System Change at which two expert panels discussed whether report cards make the grade. The first panel approached this subject through a Socratic dialogue that focused on the release of a hypothetical community hospital report card. The second panel weighed in on two research presentations related to report cards. The panelists agreed that efforts to collect and report clinical outcomes data are flawed. Even so, release of the data can help improve clinical quality and foster an environment in which health care quality information ultimately has an impact on health care decision making.

• The Evolution of Report Cards
  
• The State of Risk Adjustment
  
• Differences Among Risk Adjusters
  
• The Use of Performance Data by Employers
  
• Use Changes Along a Learning Curve

The Evolution of Report Cards

isk-adjusted hospital outcome reports have been available publicly for more than a decade. Initially, these reports were viewed as revolutionary vehicles for information on what really happens to patients at different hospitals. Do they live? Do they die? Do they get better or worse? Prior to the release of outcomes data, most hospitals were judged on the basis of their reputations, with some attention to whether they were accredited by the Joint Commission for the Accreditation of Healthcare Organizations (JCAHO). But the use of outcomes data, which began in earnest in 1986 when the Health Care Financing Administration (HCFA) launched its hospital mortality reports, was supposed to change all that by raising the bar for provider accountability.

More than 10 years later, the verdict on the report card movement is mixed. Reporting of risk-adjusted outcomes data is an inexact science at best, and some believe it is seriously flawed. Several initiatives, including HCFA’s mortality reports and Colorado’s hospital report card program, have died. But overall the number of report card activities is increasing, spurred partly by employers’ and consumers’ demands for information on health care quality and partly by hospitals’ desire to demonstrate their value in a more competitive market. Some states -- including New York, California and Pennsylvania -- have mandated outcomes reporting, thereby making it a matter of public policy.

It is unclear how all this information is being used and what impact it is having on health care markets. Panelists were confronted with a number of issues, including:

• What are the needs of the multiple audiences for outcomes data?

• How can information best be leveraged when needs differ among targeted audiences?

• What information can people use, and how should it be presented to them?

• How valid is the information presented in outcomes-based hospital report cards, and what conclusions may be drawn about quality from these report cards?

• How does the release of hospital-specific performance information affect the quality and delivery of care in communities?

Back to Top

The State of Risk Adjustment

ebate about the validity of outcomes-based information invariably focuses on the risk adjustment used to compensate for differences in health status among patients that may affect their treatment outcomes. Risk adjustment, explained Lisa I. Iezzoni, associate professor of medicine at Harvard University, helps "tease out quality-of-care differences across providers when looking at outcomes of care," and is essential to the success of any credible report card initiative.

But risk adjustment is not a simple endeavor. It may be approached in a number of ways, the selection of which may strongly influence reporting results. Before deciding on a risk-adjustment strategy, Iezzoni noted, leaders of report card initiatives must know what questions they want answered and what risks they seek to control. For example, a study in which cost is the major risk requires a different approach from a study in which the stakes center on in-hospital mortality.

The next consideration is choosing a data source. Iezzoni reviewed the pros and cons of the two primary data sources available for risk adjustment.

Administrative data, she noted, are less costly to collect and are more readily available for populations of patients. However, they contain limited clinical information, and their credibility is tarnished because of their link to payment and their inability to identify the timing of comorbid conditions.

Medical record data are much richer in clinical detail, which facilitates risk adjustment. Physicians view these data with more credibility, but they are difficult and costly to collect. In addition, the breadth of the medical record, which makes it so attractive in one respect, makes it difficult to decide which clinical data elements should be included in the risk-adjustment methodology. The risk elements selected must be widely available and without bias across all provider types, so that large and small, teaching and non-teaching hospitals may be compared fairly.

Back to Top

Differences Among Risk Adjusters

hen comes the matter of selecting from among the many commercially available severity-adjustment products. Iezzoni evaluated 14 systems by reviewing in-hospital mortality among 1,200 heart attack patients treated at 100 hospitals around the country. She discussed her findings for three of those products: MedisGroups, Disease Staging and APR-DRGs. Of the three, MedisGroups is the only system that uses clinical data.

She found that these systems varied widely in how they assigned risk to patients and in how they identified low-performing hospitals. In addition, they explained only a small percentage of the variation in outcomes among providers.

	Risk adjustment helps tease out quality-of-care differences across providers when looking at outcomes of care, and is essential to the success of any credible report card initiative. -- Liza Iezzoni

On the surface, the two administrative database systems, Disease Staging and APR-DRGs, performed better statistically than the clinical data product, MedisGroups. But this could be misleading because of the retrospective nature of administrative data products, which collect information at discharge. "The administrative database systems are ’post-dictors,’ not ’pre-dictors,’" Iezzoni said, because they take into account everything that happened to the patient during hospitalization, even when it is the result of poor care. In effect, they may mask problems of quality by "adjusting them away." MedisGroups, on the other hand, is designed to take a snapshot of the patient at admission and uses data from the first two days of hospitalization.

Based on Iezzoni’s research, the science of risk adjustment has a long way to go. That said, there are at least two compelling reasons to perform risk adjustment:

• to begin a productive dialogue on how to improve quality with physicians, who will not not believe or accept outcomes data unless they are risk-adjusted and

• to minimize physicians’ incentives to avoid high-risk patients.

Robert M. Krughoff, president of the Center for the Study of Services, was disturbed by Iezzoni’s finding that differences in risk adjustment could produce misleading or even inaccurate information about health care quality. "We could actually have people going to exactly the wrong hospitals," he noted. But because of the per-vasive pressure to control health care costs, it is more important than ever to monitor health care quality, he added. "We need to do both things," Krughoff said. "We need to report the data, and we need to continually work to improve the methodology."

Dwain L. Harper, executive director of the Cleveland Health Quality Choice Program, also believes that it is more important to achieve political consensus on risk adjustment within a given market than it is to achieve technical perfection. "It doesn’t matter what models you use or what system you use, as long as all the players agree to the methodology and agree to abide by it," he asserted.

Although some consensus around risk-adjustment and outcomes reporting appears to be building regionally in markets such as Cleveland, the national trend appears to be just the opposite, noted Jeffrey A. Rideout, vice president of quality management for Blue Cross of California. He cited the JCAHO’s new performance measurement program, which allows hospitals to select from a menu of 61 systems the JCAHO has reviewed and approved for reporting outcomes data. That level of flexibility, he said, will undermine health plans’ and employers’ efforts to discriminate among providers on the basis of quality.

Back to Top

The Use of Performance Data by Employers

hile Iezzoni’s research demonstrated the technical challenges to reporting outcomes data, a presentation by Judith H. Hibbard, professor of health policy at the University of Oregon, illustrated the difficulties of getting decision makers to use these data. Hibbard conducted a survey of large employers in four regions where hospital outcomes data are available to the public: California, New York State, Pennsylvania and metropolitan Cleveland, Ohio. Her objective was to examine how employers use performance information in their purchasing decisions.

	Some purchasers are just beginning to compare plans on customer service performance; others are further along in development and are comparing plans on some measures of clinical quality. -- Judith Hibbard

Many purchasers were not even aware of the availability of performance data in their regions. Awareness ranged from 25 percent to 83 percent across regions. With the exception of Cleveland, most purchasers did not use hospital outcomes data in their purchasing decisions. They were more likely instead to consider consumer satisfaction data and accreditation status as quality determinants.

Employers cited a number of reasons for not using hospital outcomes data:

• They did not believe the data were sufficiently reliable or timely.

• They assumed that the plans with which they did business examined hospital performance data themselves.

• The information was not presented to them in a useful fashion, particularly when their needs centered on plan selection, as opposed to hospital selection.

• They could not use the information for making purchasing decisions.

In addition, Hibbard identified several cognitive barriers to employers’ use of hospital outcomes data. Purchasing selection is a complex process that requires employers to integrate many variables into their decisions so sometimes they try to take shortcuts. For example, they may focus on only one or two variables, or they may choose to consider an indicator, such as accreditation status, that integrates and summarizes several aspects of quality into a single, easy-to-understand measure.

In situations where purchasers have to make trade-offs between cost and quality, they often choose subjectively. Hibbard said that only about 20 percent of purchasers used some kind of objective evaluation system to make trade-offs. She also noted that people generally give more weight to concrete or precise factors than to factors that are difficult to evaluate. "This suggests that when purchasers have to make trade-offs, understandable factors like access and cost may outweigh quality factors that tend to be vague and less understandable," she said. Another important role for large purchasers is educating employees about quality, Hibbard noted. Thirty-one percent of employers said they share performance information with employees, and an additional 15 percent expressed plans to do so.

Back to Top

Use Changes Along a Learning Curve

ibbard observed that the use of performance data varies among purchasers and is changing over time. "Some purchasers are just beginning to compare plans on customer service performance; others are further along in development and are comparing plans on some measures of clinical quality," she reported. Several employers had been using clinical quality indicators for a while and were gaining experience in using this information to answer their questions about quality.

Among employers that do not make extensive use of performance information, several indicated a desire or plans to do so. The challenge for the producers of this information is to promote its use by supporting the learning process, Hibbard said.

She offered several recommendations:

• Tailor quality information to purchasers’ needs.
  
• Educate purchasers to support the use of quality information.
  
• Develop clinical measures that summarize information into easily understood indicators.
  
• Develop decision aids to help purchasers use the information to achieve their goals.
  
• Upgrade information given to employees about quality of care.

Some of these recommendations may be difficult to put into action. For example, the experience of Pennsylvania’s Health Care Cost Containment Council shows that employers are not always certain themselves what their information needs are, and as they grope for answers, their demands for information escalate. The Pennsylvania council has had difficulty keeping pace with employers’ changing information requests: first for data on hospital services, then for physician-specific data, then for a quicker turnaround. The council complied with each request, and now has been asked for information on managed care plans, which it is preparing to report.

"They keep moving the bar on you and they keep indicating that when they have this particular element, they’ll use the data," says the council’s executive director, Ernest Sessa. To date, there has been little evidence that has happened.

In addition, employers’ information needs should not be equated with those of their employees. Carol Cronin, senior vice president of Health Pages, noted that while employers are pursuing quality information at the health plan level, because that is where they make their purchasing selections, consumers for the most part are "making decisions about heart surgery or breast cancer treatment in their own communities with doctors and hospitals." Many consumers have little or no choice with respect to health plans because their employers offer only a limited number -- or even just one.

Clark E. Kerr, president of Consumer First, suggested that employers will start to put outcomes data to better use when they recognize their connection to workplace productivity. As employers "see the relationship between good health care -- both mental health and physical -- and productivity, corporate America will start to reward people for increasing productivity, which will require a better understanding of and attention to quality of care."

Panelists were also concerned about the role of the news media in reporting hospital-specific outcomes data. But Harper pointed out the news media can be important allies in promoting public awareness and understanding. "The print media in Cleveland have done a tremendous job of assisting us in communicating not only the data, but the explanations of the data as well," he said.

The panelists agreed that the production and use of outcomes data is fraught with complex issues that unfold differently in different markets. Market forces are shaping these developments, although it may be only a matter of time before information on health care quality becomes a force in its own right.

	This Issue Brief is based on a seminar sponsored by the Center for Studying Health System Change, held in Alexandria, Va., on May 7, 1997. Panel One -- "Local Hospital Flunks Out": The Hospital Report Card in This Morning’s Newspaper Moderator Willis B. Goldbeck Institute for Alternative Futures Panelists James A. Block -- McManis Associates, Inc. Thomas J. Davies -- GTE Susan Dentzer -- U.S. News and World Report Norman S. Kato -- University of California at Los Angeles Medical Center Wayne R. Moon -- Blue Shield of California Keith B. Pitts -- OrNda HealthCorp John C. Rother -- American Association of Retired Persons Panel Two -- Putting Report Cards to the Test: What Research and Experience Tell Us about the Relevance of Hospital Report Cards Moderator Mark W. Legnini -- Center for Studying Health System Change Presenters Judith H. Hibbard -- University of Oregon Lisa I. Iezzoni -- Harvard University Panelists Carol A. Cronin -- Health Pages Edward L. Hannan -- State University of New York at Albany Dwain L. Harper -- Cleveland Health Quality Choice Program Clark E. Kerr -- Consumer First Robert M. Krughoff -- Center for the Study of Services Arnold S. Milstein -- Pacific Business Group on Health, William M. Mercer, Inc. Jeffrey A. Rideout -- Blue Cross of California Ernest J. Sessa -- Pennsylvania Health Care Cost Containment Council

Back to Top