Patient Reported Outcomes – Choosing Appropriate Metrics from a Still-Evolving Toolset

Patient Reported Outcomes – Choosing Appropriate Metrics from a Still-Evolving Toolset

Patient Reported Outcomes

Choosing Appropriate Metrics from a Still-Evolving Toolset

By Wes Chapman, Mike Choukas, Charles Hutchinson PhD, Steve Maker, Mario Martinez


This paper defines a set of criteria for choosing Patient Reported Outcome metrics for use in oncology, and lists five PROs the authors found suitable for further study and trial. The criteria should be useful for testing the suitability of new metrics as they become more sophisticated and more accepted in clinical care and in oncology in particular.


Patient reported outcomes (PROs) are widely recognized as an important component in measuring healthcare outcomes. The National Institutes of Health (NIH), Food and Drug Administration (FDA), and Centers for Medicare and Medicaid Services (CMS), in their various programs, have all specified the use of PROs in determining effectiveness, safety, or value of a particular clinical program, pathway, medical device, or drug. This makes clear sense; the goal of healthcare is to improve the health of patients, and only the patients themselves can tell us how well they feel, before, during, and after the treatment.

As opposed to objective measures of health (blood pressure, bone density, lung capacity, etc.), which doctors or nurses can measure and report, PROs depend on answers patients provide to subjective questions about physical, mental, and emotional health: how much pain they’re feeling, how much energy, how much anxiety. In fact, there are some conditions – gastrointestinal distress, depression – where PROs are the only available measure without additional testing. PROs also examine the effects of these feelings on functional aspects of life; for example, how much their pain has affected their work, play, social life, and so on.

PROs are obtained through short surveys, either self-administered by the patient or through an interview, and usually take the form of rating scales or yes/no questions; for example:

I have pain … Not at All – A little bit – Somewhat – Quite a bit – Very much

PROs measure a patient’s sense of health and well-being, now and over time, indicating whether a particular treatment has been effective. They can also be combined with measures of service (During this hospital stay, how often did nurses explain things in a way you could understand? Never – Sometimes – Usually – Always) to measure a patient’s experience and satisfaction with the treatment they received, and they can be aggregated with other patients’ answers to provide a measure of the total process the patient went through, as well as a rating of the hospital or clinic that provided the treatment.

                                Cartoon PRO's Paper                           
Cartoon by xkcd at

Finally, PROs can be an important source of metrics for quality improvement at every level: the individual case, the pathway, the physician, the team, and the facility. Given this level of usefulness, we should expect to find a robust set of PROs (or metrics derived from PROs, sometimes referred to as PROMs) that physicians and clinics can choose from to suit their particular needs.

Unfortunately, this doesn’t seem to be the case.

Still in Their Adolescence

The current state of PROs is best described in an expert commentary posted May 14, 2012 on the website of the National Quality Metrics Clearinghouse (NQMC):

For more than 40 years … outcomes have had a central role in how high-quality health care is conceptualized. In an era in which individualized or personalized care, patient-centered care, patient-centered medical homes, and “patient-important” outcomes are increasingly invoked as critical aims for the U.S. health care system, using patient-reported outcomes (PROs) in quality assessment and improvement schemes seems crucial.

The significance of this view has not yet, it would seem, reached any substantial level of practical application. –Kathleen N. Lorh, PhD

Dr. Lohr, a researcher with RTI International at Research Triangle Park in North Carolina, has worked in the field of healthcare quality metrics for over two decades. She is affiliated both with the PROMIS Health Organization (a nonprofit foundation that develops and administers PROs), and the Core Editorial Board for the National Quality Measures Clearinghouse and the National Guideline Clearinghouse. She bases her statement not only on her experience, but on the simple fact that “…even though NQMC has more than 360 outcome measures (as of early 2012), the Clearinghouse has very few patient-reported outcome measures – those directly salient to and reported by patients about the end results of health care.” By her count, somewhere between 20 and 40 measures at most involve PROs. She can’t be more definite because the NQMC’s search functions don’t allow a simple means to find PROs, which is, in itself, a statement about the status of PROs.

Dr. Lohr points out that thousands of PROs exist, but few of the developers have made an effort, or succeeded, in submitting their measures to the NQMC for approval and inclusion. She analyzes why this is the case, and provides a solution that revolves around the NQMC’s requirements for inclusion. Both her suggestions and the editor’s response imply that most developers of PROs would not have any trouble getting their measures approved. Yet that hasn’t been happening. Given the requirements from NIH, FDA, and CMS, this seems remarkable.

Barriers to Endorsement

It’s significant to note that CMS has endorsed only one PRO measurement tool, the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS), which is actually far more focused on hospital services than it is on patient health and well-being. Questions on the HCAHPS range from “how often did the nurses treat you with courtesy and respect” to “how often was the area around your room quiet at night”. It does not ask if the patient is currently in pain, anxious, nauseous, able to walk or work. It does not compare the patient’s current state with their previous state, nor with their expectation of how they should feel after going through treatment. It doesn’t measure outcomes, it measures process, and a limited part of the process at that: just a few variables that are universal to every hospital experience, heavily weighted toward hospitality rather than clinical treatment.

Why is this significant? Because the HCAHPS is now a key piece in the payment formula for Medicare and Medicaid patients. Hospitals and clinics are driven to use the HCAHPS in order to be paid, not because it’s an effective tool in quality improvement. Until CMS requires other PROs for a wider range of situations and more related to the health state of the patient, there will be less incentive for the industry to start using them. But this is not the only barrier.

Most PROs Were Not Designed for Clinical Care

Many PROs, if not most, were developed for use in clinical trials, not for use in standard clinical care. Using PROs in trials can be vitally important, of course, helping to determine if new treatment pathways are beneficial from the patient’s perspective. Because these PROs have been validated in the clinical trial arena, however, some would argue that they are not reliable quality metrics for use in clinical treatment. This ignores the fact that the patient’s experience is very much the same in both settings, and the effects of the disease vary only to the extent that treatment pathways vary. In oncology, there will still be the same array of symptoms to deal with – pain, nausea, anxiety, depression, fatigue, sleeplessness, and so on – with slight variations depending on the specific cancer. And the questions you want to ask the patient about their physical, mental, and emotional state are the same.

Most PROs Are Not Disease Specific

The large majority of PROs are designed to assess the general state of a patient’s well-being. This makes sense; the health effects that PROs attempt to measure are caused by most diseases. A migraine, for example, is far more than a pain in the head. But it is true that certain diseases have a spectrum of effects that are not all covered in a general health assessment survey. Cancer and its treatment can cause pain, nausea, weakness, vertigo, confusion, weight loss, hair loss, fatigue, sleeplessness, anxiety, depression, and more. However, the extreme cases do seem to be covered. There are PROs for general cancer and some specific cancers, as well as for headaches, asthma, hepatitis, HIV, and more. And these tools can be combined with a general health PRO to generate a more compete picture of health and well-being.

Many PROs Conflate Process with Outcome

This seems to be particularly true of hospital rating tools, such as the HCAHPS and the many commercially offered tools that have the CAHPS as their core metric set. As we said above, these tools include quite a few measures of hospitality and service during treatment, rather than measures of the quality of clinical pathways and their before-and-after effects on the patient. Or they mix the measures in such a way that it is difficult to separate hospitality from treatment and process from outcome. While it is undoubtedly true that the total healthcare experience, from ease of parking to post-discharge follow-up, has an effect on the patient’s well-being and satisfaction, providers are rightfully concerned that too much emphasis can be placed on factors that are only marginally important to health outcomes.

PROs Imply Criticism of Doctors and Clinics

Healthcare providers are not always happy to see their work measured in such a personal and transparent way. As one doctor put it:

“In medicine, we are used to confronting failure … What we’re not used to doing is comparing our records of success and failure with those of our peers … the truth is that we have had no reliable evidence about whether we’re as good as we think we are.” –Atul Guwande, M.D.; Better: A surgeon’s notes on performance (p.207); 2007, Picdor/Henry Holt & Company.

PROs provide that evidence, and they have special weight, because they are essentially spoken in the patient’s own words. Doctors and the facilities where they work not only cringe at the potential criticism, they worry that a poor rating will drive away business. Dr. Gawande’s book shows at least one example where that doesn’t seem to be the case, and where transparency has led to quality improvement in all of the clinics being rated, including those already at the high end of the curve. It’s important to note, though, that the PROs must be appropriate to the situation and, to state the key point of any quality metric, they must be analyzed, understood, and acted on. Doctors, nurses, and other staff must be educated about the benefits to both patient and provider that accrue from the quality improvement gains made possible with PROs; not just higher ratings, but also better treatment pathways, more patient involvement, better palliation, and better outcomes all around.

Criteria for Choosing the Right PROs

In a previous paper, we outlined a taxonomy of organizations that endorsed clinical pathways and metrics in oncology. Our selection criteria there was based on the Anna Karenina Principle, which states that the key organizations will be the only ones that meet a specific set of must-have characteristics. PROs require a less restrictive approach. The range of uses to which they are put, along with their current evolving state of development and acceptance in healthcare, is such that no one set of criteria would identify PRO instruments that fit every application. That said, we have been able to define a set of five characteristics that apply to many PROs in varying degrees. In some cases, it’s a loose fit, but the importance of each criterion to any given application can be considered when making choices.

Our five characteristics are:

1)      Applicable to Oncology – The questions cover a range of negative symptoms associated with cancers, as well as measures of general health and functionality. Some PROs, such as the FACT-G, are designed specifically for oncology and rate very highly here. Others, such as the SF-36 are general health assessment tools, but do ask about pain, anxiety, and other symptoms applicable to Oncology.

2)      Broadly used – The PRO instruments have been used in many institutions or trials. (Since so few have been endorsed by CMS, AHRQ, or NQF, this provides a reasonable measure of acceptability and potential for endorsement.) It can be difficult to determine if a particular PRO instrument is widely used in clinical care without relying on unsupported claims made by vendors.

3)      Well validated – The PRO instrument tests well for both validity (the measure is associated with what it purports to measure) and reliability (the measure is free from random error). Note that, in some cases, the only statements of validity and reliability that are easy to find come from the vendor of the instrument.

4)      Easy to use ­– The survey is brief and easy to understand and complete by a person under the stress of treatment. The FACT-G, for example, has 28 questions, while the Symptom Impact Profile has over 100, and a long set of detailed instructions. Alternative methods of administration, such as by interview, by phone, with assistance, or via computer (tablet, online, etc.) can simplify a long instrument or complicated set of questions (e.g., if the answer is Yes, jump to question 13).

5)      Clinically Useful – That is, the measure can be used to immediately adjust treatment to improve the patient’s state of health. In an Anna Karenina universe, the HCAHPS would not be welcome in the happy family of PROs.

A Selection of PROs for Oncology ACOs

Our projects require PROs suitable for use in accountable care organizations (ACOs) focused on clinical oncology. Using our five criteria, we have selected four PROs that fit the critera fairly well, and have decided to include one outlier accepted due to regulatory need.


Hospital Consumer Assessment of Healthcare Providers and Systems

This is the only PRO instrument endorsed by CMS; it is required in the Medicare Shared Savings Program. There are forty-five approved vendors/administrators, and more that are not officially approved. It scores very low on many of our criteria and would not be included if not for the CMS approval. It is self-administered only after discharge (via mail) or is taken by phone interview.


Short-Form-36 Health Survey

This survey is not oncology specific but has been very broadly used (with 11,000 listings in PUBMED for the search string “SF-36”). It has a fixed set of 36 questions, with shorter versions that have 12 and 8 questions (the SF-12 and SF-8), and a companion inventory that focuses on pain. It can be self-administered on paper or computer or taken via interview. The SF-36 was originally developed by the Rand Corporation, which sold the commercial rights. Rand still makes the original version available as a free, public domain instrument.


Functional Assessment of Cancer Therapy–General

The FACT-G was developed for use during treatment rather than clinical trials. It is applicable to all cancer tumors (though only three cancers were involved in its initial development: breast, lung, and colorectal). The FACT-G is a core set of metrics, with many different add-on sets designed for negative health affects in various cancers. It is brief, self-administered on paper (computer-based versions are being developed, according to the steward organization), or taken by interview (p2p or phone).


Patient Reported Outcomes Measurement Information System

PROMIS is a government program, funded by NIH, but not yet officially endorsed by CMS, AHRQ, or NQF. It is not oncology-specific, but it is very flexible; the user can select from existing core PRO sets and individual questions, as well as add questions of their own creation. It is free and hosted on the web, for use by researchers and clinicians. Users have their own secure databases, where they develop the sets they wish to use. The surveys can be self-administered on paper or online and also taken via interview (p2p or phone). Brevity and ease of use would depend on how customized you make it. It is a relatively new instrument but claims that the core set is well validated.


M.D. Anderson Symptom Inventory

The MDASI is oncology-specific and was developed for clinical and research use, with many subsets for different cancers. It is brief and can be self-administered (paper) or taken by interview (p2p or phone). It is relatively new (2000), but appears to be well validated. It is difficult to assess how broadly it has been used. (Searching PUBMED for “MDASI” reveals only 51 articles, and there are no hard data available that indicate the breadth of use of any PRO in clinical care.)


Our selection criteria have given us a reasonable start; we have several useful PRO instruments to explore further. Dr. Lohr’s expert opinion is certainly true – the developers and stewards of PROs should make more effort to have them approved by NQMC and NQF, and those organizations should do more to facilitate the process. It would be a great benefit to hospitals, ACOs, and other providers to have a list of endorsed PRO metrics from which to choose. There are too many sets and too many vendors competing for the top hits on Google, and the most effective at cadging links are not necessarily the organizations most oriented to patient health and quality improvement over operations and ratings.

It is also true the CMS must make an effort to endorse and require PROs that are more focused on quality improvement in both patient health outcomes and clinical pathway processes. The financial carrot and stick that CMS provides cannot be ignored in the drive to move PROs to a more mature state of development. This is underscored by a study published on February 12, 2012 in the Archives of Internal Medicine entitled The Cost of Satisfaction: A National Study of Patient Satisfaction, Health Care Utilization, Expenditures, and Mortality. (Joshua J. Fenton, MD, MPH; Anthony F. Jerant, MD; Klea D. Bertakis, MD, MPH; Peter Franks, MD; Arch Intern Med. 2012;172(5):405-411. doi:10.1001/archinternmed.2011.1662.

Noting that “…the relationship between patient satisfaction and health care utilization, expenditures, and outcomes remains ill defined…,” the authors conducted a study of seven years’ worth of data from the national Medical Expenditure Panel Survey. What they found was that “…higher patient satisfaction was associated with less emergency department use but with greater inpatient use, higher overall health care and prescription drug expenditures, and increased mortality.” In fact, the highest patient satisfaction quartile was 26% more likely to die compared to the lowest quartile.

This bald statistic has been seized on by some bloggers as a clear sign that PROs – at least those related to patient satisfaction – do not indicate healthcare quality. That may be true, but we can’t make that claim without first trying to fully understand the results of the study – as we would with any quality metric before acting on it. There are many questions we must ask. Were the survey questions focused on hospitality and patient experience or on health status? Were the results consistent across provider? Were satisfied patients more or less involved in the decisions regarding their treatment? Are highly satisfied patients more complacent, and thus more likely to accept whatever is offered? Are they more likely to accept their fate and not struggle against their disease? The fact is that these results could indeed show that satisfaction is a powerful tool for improving healthcare quality if we can be sure we are reading them correctly. Then we can design an appropriate adjustment to our treatments so that what we now call satisfaction is no longer an indicator of increased mortality. Or we might find that we simply need a better tool for measuring satisfaction; for example, a tool that measures health states not only before, during, and after treatment, but also against the patient’s expectation of what the outcomes should be.

In Better, Dr. Gawande describes a visit between a cystic fibrosis patient and doctor, the head of the CF clinic with the best patient outcomes by far in the U.S. The patient’s scores on several important measures had dropped, and the doctor was trying to discover why. He did what doctors always do in a one-on-one clinical situation: He looked at the scores and then he started to ask questions. In this case, he listened intently and asked more questions, until he uncovered where the patient had strayed from her treatment regime. In other words, the doctor was using PROs, made up on the fly for the specific disease, patient, and treatment pathway.

The patient, a teenager going through all of the changes and rebellions that that age brings on, balked at the discipline the doctor was asking of her, and he replied, “We’ve failed … it’s important to acknowledge when we’ve failed.” With that acknowledgement – and with both of them sharing the failure – they began to work out the adjustments she needed to make to get back on track with her treatment.

A PRO survey cannot have that level of sensitivity or insight, but a well-designed survey used in the appropriate situation can reveal when a treatment pathway is failing, and a careful analysis of the results can lead to measurable quality improvement, if not perfection.

Wes Chapman
Written by Wes Chapman

1 Comment responses

  1. Avatar
    September 27, 2012

    Any time we measure, we get better results.