Detailed Overview of

 

Clinical Practice Improvement

 

 

 

Introduction

 

This overview of Clinical Practice Improvement discusses:

 

·        Clinical Practice Improvement

-         Study Design (including patient, medical care process, and outcome factors)

-         Analytic Methods

 

·        How Clinical Practice Improvement Goes Beyond

-         Outcomes Research

-         Guidelines Development

 

·        Clinical Practice Improvement and Research-Based Dynamic Protocols

 

·        How Clinical Practice Improvement Differs from Randomized Controlled Trials

 

·        Advantages of Clinical Practice Improvement

 

                        Clinical Practice Improvement (CPI) is a study methodology designed to develop analytically based protocols to achieve desirable outcomes at the lowest essential cost over the continuum of care.1  Several elements of the Clinical Practice Improvement approach make it attractive to clinicians. 

           

                        First, it is a scientific "bottom-up" approach that places accountability for practice improvement and outcomes with clinicians.  Clinicians are not told to follow a guideline or protocol developed by others, but instead collect data on outcomes, on treatments, and on patient signs and symptoms that support practice change. Clinical Practice Improvement supports caregivers in making their own decisions about optimal care on the basis of objective statistical evidence gathered in the routine, everyday practice of medicine.                        

           

                        Second, Clinical Practice Improvement measurement encompasses a comprehensive view of the care management process: patient characteristics, process steps, and outcomes.  All three classes of data are considered simultaneously. This comprehensive measurement framework provides a basis for meaningful analyses of significant associations, as well as relationships between process and outcome.

           

                        Third, the Clinical Practice Improvement methodology focuses on application.  There is a continual emphasis on factors that can be implemented to improve outcomes and the process to achieve these results.  This focus on implementation guides who is involved in the design, what data are collected, what questions are answered during analyses, and who designs the protocols or improvements in practice.

 

 

Clinical Practice Improvement Study Design

           

                        A Clinical Practice Improvement study design includes measures of patient factors (physiologic severity of illness and psychosocial derangements presented at each visit or at each admission), medical care process factors (e.g., medications, treatments), and outcome factors.  It presents the resulting associations to clinicians, so they can evaluate objectively the effects of the treatments they give to similarly ill patients.  Without all three types of data from the care management process (e.g., if one has only process and outcome data, but not detailed patient data), clinicians cannot tell if the outcomes achieved are due to the process steps or to differences in patient severity levels; see Exhibit 1.

 

Exhibit 1.  Three Essential Components for a Clinical Practice Improvement Study

 

                        Improve/Standardize

Process Factors

-          Management Strategies

-          Interventions

-          Medications
 
 

 


                                                                                                Measure:

Outcomes

-          Clinical

-          Health Status

-          Cost/LOS/Encounters
 
                                                                                               

 

                        Control for:

 

 

 

 

 

 

 

 

 


Factors in A Clinical Practice Improvement Study Design

 

Patient Factors. Patient factors are key characteristics of the population: demographics, specific indications for treatment and severity of illness, psychosocial factors, etc.  Within a well-defined, similarly ill patient group, one would expect that care processes of equal effectiveness would result in similar outcomes of care.  To have enough detail describing patients and their needs so that clinicians will agree to stabilize their processes of care, one usually requires disease-specific physiologic data, such as those contained in the inpatient and outpatient components of the Comprehensive Severity Index (CSI®).2-10  If a detailed physiologic severity system is not available for all of a patient’s diseases, then the Clinical Practice Improvement multidisciplinary team determines what data are needed to define patients with similar levels of illness, so that a decidable and executable treatment protocol can be developed, based on patient signs and symptoms.

           

Medical Care Process Factors.  A process of care is a sequence of linked, usually sequential, steps designed to cause a set of desired medical outcomes to occur.  The goal is to find a measurable factor that describes each major process step.  Examples include which drugs are dispensed, how often prescriptions are filled, what dose is used, how is a ventilator set, etc.  A data collection instrument records the process steps in detail.

 

Outcome Factors.  Processes of care should be designed to achieve specific medical outcomes.  Among the outcomes commonly assessed in data collection instruments are diagnosis-specific complications, diagnosis-specific long-term medical outcomes (which may be assessed by both clinicians and patients), patient functional status, patient satisfaction, and cost.  Outcome factors may be thought of as analogs of the assessment endpoints in a randomized controlled trial.

                       

                        The ultimate goal of a Clinical Practice Improvement study is to help clinicians improve delivery of care and establish dynamic systems of improvement to achieve medical outcomes that are:

 

·                    improved over those achieved by the clinician group before the study and at the least necessary cost; or

 

·                    substantially the same as those achieved by the clinician group before the study, but at lower cost.

 

 

Analytic  Methods

 

A Clinical Practice Improvement study database is designed to measure patient characteristics, processes of care, and patient outcomes, and it can include many variables.  When more than three or four patient and process (independent) variables must be taken into account, multiple regression analyses can be used to model the effects of these factors on the outcome (dependent) variables.  Multivariate statistical methods allow comparisons of alternative treatments while controlling for other variables that may be driving observed differences between the outcomes of the treatments.  These statistical methods allow the researcher to examine relationships far more complex than those defined using only one explanatory variable at a time.  The coefficients of the independent variables in the regression equations identify key process steps that, when controlling for patient factors, lead to better outcomes.

 

 

Clinical Practice Improvement Goes Beyond Outcomes Research

 

Outcomes research typically uses large, existing claims databases to find outcome failures, often identified as poor outcomes beyond some statistical threshold, e.g., high mortality rates.  But most outcomes research does not lead to practice improvement because:

 

·        outcome failures are not scientifically related to detailed process steps that are under a practitioner’s control, so it is unclear how to improve the outcome; and

 

·        patients are described only by diagnosis codes, so their severity of illness is not controlled for.

 

In addition, concerns about the completeness, accuracy, and relevance of large claims databases raise questions about their appropriateness as a basis for health services research, performance monitoring, and inspiring changes in clinical practice.  A variety of initiatives have focused on means to fill in missing data, validate and improve coding of clinical and other information, add information (e.g., death records), and develop methods to adjust comparisons for differences in severity of patient conditions.  However, a fundamental problem remains: data collected for one purpose (e.g., claims administration) may not be useful for other purposes (e.g., outcomes research) if they lack reliable information about patient medical factors and medical care processes.

 

 

Clinical Practice Improvement Goes Beyond Guidelines

 

Because most clinical practices have no firm basis in published scientific research, developers of clinical guidelines often resort to expert consensus.  But expert consensus is an inexact tool even when generated with formal methods.  Different consensus groups have different goals and use different techniques.  They often develop different, even conflicting, guidelines on the same topic.11-13  Within a single consensus panel, the experts often disagree, and their assessments change when guidelines developed in a theoretical setting are applied to real patients.14  Perhaps most troubling, physician experts show wide disagreements when they assess underlying probabilities essential to consensus judgments.15-17  For example, Eddy asked gastrointestinal surgeons to assess the probability of a particular outcome for a well-defined group of patients within a specific time period after surgery.  Correctly assessing the likely outcome was essential to determining if the procedure was appropriate.  The surgeons’ assessments ranged from zero to 100 percent.18

                       

                        Most of the effort to develop guidelines is characterized by two weaknesses that hamper their relevance to local practice reform: 

 

·        guidelines are developed nationally or centrally, based on expert consensus and literature review/synthesis, and

 

·        guidelines are too general or inconclusive to be useful to clinicians. 

 

Thus, clinicians are unwilling to follow many current guidelines. 

 

The U.S. Agency for Health Care Policy and Research now favors "evidence-based" methods of guideline development, rather than consensus-based methods.  However, the patient populations the "evidence" comes from (usually randomized controlled trials) are different from those in which local translations of the guideline will be used.  Since evidence-based guidelines must be all things to all people, they are often encyclopedic and equivocal.  They are not decidable and executable at the local level and do not have credibility with clinicians.

 

Clinical Practice Improvement

 and Research-Based Dynamic Protocols

 

                        A research-based dynamic protocol is:

 

·        decidable and executable, i.e., it has specific process steps to follow based on deviations of a patient’s signs and symptoms from normal values;

 

·        based on analyses of data; and

 

·        developed in stages by a group of clinicians using Clinical Practice Improvement methodology. 

 

                        The goal of most Clinical Practice Improvement studies is to help clinicians produce these research-based dynamic protocols, i.e., protocols based on statistical findings that show the specified process steps that are associated with better outcomes. 

                       

                        There are several factors to consider for successful implementation of a research-based protocol:

 

·        awareness of the need for practice evaluation and improvement;

 

·        participation of a core group of clinicians throughout the clinical practice improvement planning, analyses, and implementation phases;

 

·        team accountability for process improvement and outcome;

 

·        "ownership" of protocol, based on team confidence in the data and analyses performed;

 

·        availability of resources for education, staff support, data collection, analyses, and reports on the results of implementing the research-based dynamic protocol.

 

·        a system to evaluate and modify the research-based dynamic protocol; and

 

·        on-going communication.

 

                        An important feature of the protocol implementation process is that if a clinician opts not to follow a step in the protocol, he or she gives substantive reasons for doing so, and the corresponding protocol element is automatically placed on the agenda for the next Clinical Practice Improvement team meeting.  The team always starts from the assumption that the protocol is not correct with regard to the clinical point under discussion.  The reasoning is that, if the protocol were correct, then the clinician would follow it.  Clinicians who disagree with the protocol step have an opportunity to present their reasoning to the team, in the context of a specific case, so the group can either modify the protocol step or reach consensus that the protocol, as written, does represent best practice.

 

Clinical Practice Improvement Differs

 From Randomized Controlled Trials

                       

                        The randomized controlled trial (RCT) has a long history as the gold standard for establishing causality in scientific research.  Randomization to diminish potential selection bias and strict control of the intervention of interest are important tools for scientists of all types.  However, the use of RCTs has been limited in some domains of inquiry because of problems such as:

 

·        ethical or practical inability to randomize patients or to control the specificity of the intervention to be studied;

 

·        prohibitive cost when cell sizes or samples are extremely large;

 

·        exclusion of large numbers of individuals who do not meet strict inclusion criteria, since one does not want the outcome of the study to be influenced by extraneous factors.  Therefore, patients with secondary problems or more severe disease are often rejected from the trial.  Only a small percentage of patients—usually 10 to 15 percent—are eligible for a trial.  The idea is to eliminate all patients whose characteristics might adversely affect or bias the outcome of the comparison between the treatment and the control arm; and

 

·        potential selection bias from non-participation in studies where limited benefit for specific patients or health plans can be identified prospectively.

 

                        The alternative study designs used in Clinical Practice Improvement provide a pragmatic balance of study overhead, clinician participation, rapid patient accrual, and the need for timely information vs. potential bias.  Achieving this balance is especially important when examining operational process-of-care factors (in contrast to testing new treatments) and when permanent data collection instruments routinely track patient and process factors, so that invalid inferences are likely to be found and corrected over time.

           

                        RCTs use a protocol document to create an artificial practice environment that allows for valid statistical inference.  While that structure eliminates practice variation, it usually covers a very limited subset of patients and practices.  Clinical Practice Improvement addresses the same issues—practice variation and valid statistical inference—from another point of view.  It measures process variation, then eliminates it through a combination of statistical analysis, consensus, and feedback.  Under a Clinical Practice Improvement protocol, valid statistical inference is possible, because groups of similar patients receive the same treatment.  RCTs also tend to be limited in time; in most circumstances, they explicitly modify clinician behavior only for the duration of a study and only for the individuals directly involved in the trial.  In contrast, Clinical Practice Improvement establishes a permanent feedback loop aimed at all clinicians in an institution.  It integrates research into daily practice, giving individual clinicians the information necessary to understand and modify their own activities at a detailed, operational level.  Clinical Practice Improvement analyses help the team evaluate current practices and use the results to develop fact-based improvements.  Changes to the process of care rest on clinical data rather than on clinical opinion.

           

                        Conducting an RCT to examine multiple disease conditions across multiple system and process factors is very expensive, perhaps even prohibitively expensive, as it often requires an elaborate bureaucracy to coordinate care and collect data as an add-on layer.  For example, the Medical Outcomes Study and the Health Insurance Experiment conducted by RAND in the 1980s cost sponsor organizations more than $35 million.  The Infant Health and Development Program, a smaller, randomized trial with only two arms (intervention vs. non-intervention) for 1,000 low-birth-weight infants in nine cities, cost more than $19 million for the first six years of data.  The SUPPORT study to determine the best practices for patients to die with dignity and without pain cost $28 million.  Each of these studies had smaller sample sizes and fewer cells than the Managed Care Outcomes Project, a Clinical Practice Improvement study in which multiple sites and multiple levels of intervention (degree of formulary restriction, gatekeeper restrictiveness, visit co-pay, etc.) were examined.19, 20  No study since the 1980s has had a budget large enough to accommodate a sample size as large as the 12,997 patients in the Managed Care Outcomes Project, which cost about $500,000.

           

                        The design of the Managed Care Outcomes Project also allowed the inclusion of large numbers of patients likely to have been excluded from an RCT, thus improving generalizability and external validity.  The ability to measure severity and control for confounding variables across multiple domains permits identification of associations rather than causality; however, the results of sensitivity analyses and the concurrence of findings with other research can help to determine whether the identified associations are real and are relevant to existing practices.21, 22  Finally, the Clinical Practice Improvement study design in the Managed Care Outcomes Project also resulted in an extremely low rate of attrition over time, thus avoiding a persistent problem with RCTs in health services research.

 

                        Recent literature has supported the use of well-designed observational studies, such as those using CPI methodology, to discover what works best in medicine.  Two studies appearing in the June 22, 2000, issue of the New England Journal of Medicine found that treatment effects from observational studies and randomized controlled trials were remarkably similar.23-24  Both studies concluded that they found little evidence that estimates of treatment effects in well-designed observational studies were either consistently larger than or qualitatively different from those obtained in randomized controlled trials.

 

 

Advantages of Clinical Practice Improvement

 

A key advantage of Clinical Practice Improvement methodology is the naturalistic view of medical treatment that is provided by retrospective data recorded routinely by medical providers.  This view is critical to determine implications of treatment alternatives.  In everyday practice, patients are assigned to different treatments based on the provider’s medical judgment, patient compliance is not artificially influenced, and monitoring of results is based on the provider’s need for information about how a patient is doing.  All these factors can impact the effectiveness of medical treatment. 

           

This retrospective view is in direct contrast to that of traditional randomized controlled trials.  Because their participants are screened, selected, and subjected to scrutiny and intervention beyond that occurring in everyday treatment, RCTs sometimes report results that are not broadly applicable in everyday medical treatment.

 

A second key advantage of Clinical Practice Improvement study methodology is cost.  Using existing data from medical records and computerized databases is generally much less costly than implementing a prospective RCT.  Other advantages of retrospective data include the large number of observations that can be available for analysis and the usefulness of the data for hypothesis generation and refinement.  Observational studies do not scientifically prove the causality of any underlying relationships, but they can point to hypotheses that can be clinically evaluated.

 

 

Clinical Practice Improvement in the Future

 

Today, data needed to conduct a Clinical Practice Improvement study are typically abstracted by hand from existing paper medical records.  In the future, most hospitals will use computerized clinical information systems (CIS).  Then, rather than relying on labor-intensive manual data abstraction, the needed patient, process, and outcome data can be found electronically in the hospitals’ CIS.  The efficiency and logistics of this new data acquisition modality will make it easier and less costly to conduct iterative Clinical Practice Improvement studies to determine best practices.  Also, the resulting research-based dynamic protocols can be programmed into hospitals’ CIS to flag for clinicians the appropriate protocol steps for a specific combination of patient signs and symptoms.  This should result in more consistent implementation of protocol steps than without these flags.

 

 

Summary and Conclusions

 

Clinical Practice Improvement studies involve a rigorous form of quasi-experimental research.  Quasi-experimental designs cover a variety of strategies that need not include a control group or random assignment.  Although they are weaker than RCTs on internal validity, Clinical Practice Improvement studies better represent actual conditions of practice, and they usually cost less and take less time.  Because they do not insist on homogeneous patient populations, they can include patients with comorbidities or complications.  To avoid confounding the link between the experimental intervention and patient outcomes, they measure relevant patient characteristics using severity assessment tools and statistically adjust for differences in experimental and comparison groups.  Further, they accommodate departures from rigid treatment protocols by carefully monitoring and measuring actual treatments; they then use these data in the statistical analysis.  Because this approach does not disqualify large numbers of patients, it facilitates generation of the number of cases needed for comparisons.  Using multiple regression and other statistical techniques, researchers test which process steps are associated with desirable quality and cost outcomes for different kinds of patients.

 

Although Clinical Practice Improvement studies tend to focus on short-term outcomes, these outcomes include effects that are noticeable and important to patients rather than only those that are physiologically measurable through laboratory or other tests.  Clinical Practice Improvement studies are designed to be replicated easily so they can be undertaken at multiple sites.

 

Methodology alternatives such as Clinical Practice Improvement do not replace the RCT, but rather provide additional sources of systematic outcomes information that improve on the anecdotal and informal knowledge base that underlies much of clinical practice.  Clinical Practice Improvement studies used by clinical teams have enormous power to enable health care providers, managed care organizations, and individuals to evaluate current practice and improve clinical decision making.

 

Source

 

            This overview is based on Chapter 1, Introduction: Overview of Clinical Practice Improvement, by Susan D. Horn, Ph.D., from the book, Clinical Practice Improvement Methodology: Implementation and Evaluation, edited by Dr. Horn, published in 1997 by Faulkner & Gray, New York, NY, and available from F&G at 1-800-535-8403.

           

This book is a source of information for clinicians and administrators who would like to apply the principles and techniques of Clinical Practice Improvement in their own settings.  Its primary audience includes physicians; nurses; hospital, clinic, and managed care plan administrators; and health policy makers who want to use the scientific method to continuously improve the quality of patient care.  It can also serve as a basic text for training programs such as the Advanced Seminar in Clinical Practice Improvement, conducted by the Institute for Clinical Outcomes Research.

 

The book is organized into four sections.  The first section discusses why Clinical Practice Improvement is useful to integrated health systems, clinician groups, employers, and government agencies.

 

The second section deals with the core concepts and steps in the Clinical Practice Improvement process, including the tools used.  It has attempted to convey sufficient details about the philosophy and methods to enable readers to initiate and carry out studies at their own practice sites. 

 

The third section contains 12 case studies that used Clinical Practice Improvement methods.  The purpose in including these examples is twofold:

 

1. to illustrate the diversity of settings in which Clinical Practice Improvement has been applied, and

 

2. to give more examples of how the tools described earlier can be used in practice. 

 

These studies will help inspire readers to set up study teams in their own practice settings.

           

The fourth section addresses some practical issues that often arise in the course of initiating and sustaining a Clinical Practice Improvement program: how does one lead and manage this new technology?  What are the organizational factors most likely to determine success or failure of such a program?  What are the roles of physicians, managers, support staff, health care executives, and trustees?  What cultural changes are required?  What are the most common impediments to success, and how can they be overcome?  How can Clinical Practice Improvement be used in wellness programs?

 

Many of the book's chapter authors are clinicians.  This is no accident, since Clinical Practice Improvement is really about clinicians taking back control of care for their patients from insurers and administrators.  By adopting these principles, clinicians, other members of the care team, support staff, and administrators can collaborate to reduce costs and improve outcomes for their patients.  This is managed care in its highest sense—no longer should it refer to interference by outside parties with little or no knowledge of the individual patient.  Instead, decisions about optimal care should be made locally, based on hard, objective, statistical evidence gathered in the routine, everyday practice of medicine.

 

 

References

 

1.      Horn SD and Hopkins DSP.  Clinical Practice Improvement: A New Technology for Developing Cost-Effective Quality Care.  New York: Faulkner & Gray, 1994.

 

2.                  Ibid.

 

3.                  Horn SD.  “Clinical Practice Improvement: Improving Quality and Decreasing Cost in Managed Care.”  Medical Interface July 1995: 60-70.

 

4.                  Horn SD, Sharkey PD, and Levy R.  “A Managed Care Pharmacoeconomic Research Model Based on the Managed Care Outcomes Project.”  Journal of Pharmacy Practice 8(4) 1995: 172-177.

 

5.                  Horn SD, Sharkey PD, Tracy DM, Horn CE, James B, and Goodwin F.  “Intended and Unintended Consequences of HMO Cost Containment Strategies: Results From the Managed Care Outcomes Project.”  American Journal of Managed Care 2(3) 1996: 253-264.

 

6.                  Horn SD, Sharkey PD, and Gassaway J.  “Managed Care Outcomes Project: Study Design, Baseline Patient Characteristics, and Outcome Measures.”  American Journal of Managed Care 2(3) 1996.

 

7.                  Horn SD, Buckle JM, and Carver CM.  “The Ambulatory Severity Index: Development of an Ambulatory Case Mix System.”  Journal of Ambulatory Care Management 11 1988: 53-62.

 

8.                  Horn SD, Sharkey PD, Buckle JM, Backofen JE, Averill RF, and Horn RA.  “The Relationship Between Severity of Illness and Hospital Length of Stay and Mortality.”  Medical Care 29 1991: 305-317.

 

9.                  Averill RF, McGuire TE, Manning BE, Fowler DA, Horn SD, Dickson PS, Coye MJ, Knowlton DL, and Bender JA.  “A Study of the Relationship Between Severity of Illness and Hospital Cost in New Jersey Hospitals.”  Health Services Research 27(5) 1992: 587-617.

 

10.              Iezzoni LI.  Risk Adjustment for Measuring Health Care Outcomes.  Ann Arbor, MI: Health Administration Press, 1994.

 

11.              Kellie SE and Kelly JY.  “Medicare Peer Review Organization Preprocedure Review Criteria.”  Journal of the American Medical Association 265(10) 1991:1265-1270.

 

12.              Audet AM, Greenfield S, and Field M.  “Medical Practice Guidelines: Current Activities and Future Directions.”  Annals of Internal Medicine 113(9) 1990:709-714.

 

13.              Leape LL, Park RE, Kahan JP, and Brook RN.  “Group Judgments of Appropriateness:  The Effect of Panel Composition.”  Quality Assurance in Health Care 4(2) 1992: 151-159.

 

14.              Park RE, et al.  “Physician Ratings of Appropriate Indications for Three Procedures:  Theoretical Indications vs. Indications Used in Practice.”  American Journal of Public Health 79(4)1989: 445-447.

 

15.              Eddy DM.  A Manual for Assessing Health Practices and Designing Practice Policies.  Philadelphia:  The American College of Physicians, 1992.

 

16.              Eddy DM.  “Variations in Physician Practice: The Role of Uncertainty.”  Health Affairs 3 1984: 74.

 

17.              O’Connor GT, Plume SK, Beck JT, et al.  “What Are My Chances?  It Depends on Whom You Ask.  The Choice of a Prosthetic Heart Valve.”  Journal of Medical Decision Making  8(4) 1988: 341.

 

18.              Eddy, "Variations," p. 84.

 

19.              Horn, et al.  “Intended and Unintended Consequences.”

 

20.              Horn, et al.  “Managed Care Outcomes Project.”

 

21.              Magi D, Douglas JM Jr., and Schwartz JS.  “Doxycycline Compared with Azithromycin for Treating Women with Genital Chlamydia Trachomastis Infections: An Incremental Cost-Effectiveness Analysis.”  Annals of Internal Medicine 124(4) 1996: 389-399.

 

22.                          Pestotnik SL, Classen DC, Evans RS, and Burke JP.  “Implementing Antibiotic Practice Guidelines through Computer-Assisted Decision Support: Clinical and Financial Outcomes.”  Annals of Internal Medicine 124(10) 1996: 884-890.

 

23.  Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials.  NEJM 2000;342:1878-86 (June 22, 2000).

 

24.  Concato J, Shah N, Horwitz RI.  Randomized, controlled trials, observational studies, and the hierarchy of research designs. NEJM 2000;342:1887-92 (June 22, 2000).

 

25.