Description of the
CSI^® Software System
Comprehensive Severity Index

Introduction

The Comprehensive Severity Index was developed by Susan D. Horn, Ph.D., senior scientist at International Severity Information Systems, Inc. (ISIS), beginning in 1982 with the help of more than 150 physicians at The Johns Hopkins University Medical School. Dr. Horn and her colleagues at ISIS, in collaboration with medical specialists across the United States, have continually expanded and improved the system, which is now available in the CSI software system.¹

CSI classifies patients according to their severity of illness, defined as the physiologic complexity comprising the extent and interactions of a patient's diseases presented to medical personnel. The severity of illness scores are based on the degree of abnormality of individual signs and symptoms of a patient's disease or diseases. Thus, they are based on physiologic measures of a patient's condition, not just on diagnostic information (ICD-9-CM coding). The more abnormal the signs and symptoms, the higher the score, with Level 4 signs and symptoms being catastrophic, life-threatening, or likely to result in organ failure.

The CSI Software System Is Comprehensive

CSI is comprehensive: it includes disease-specific, setting-specific, and age-specific criteria and weighting systems to measure severity for inpatients, ambulatory patients, hospice, rehabilitation, and long-term care patients, for all diagnoses and ages of patients, adult and pediatric. It can be used for patients with any ICD-9-CM diagnostic and procedure codes and includes severity criteria for all diagnoses. CSI can be used to predict any severity-dependent outcome, including mortality, morbidity (change in severity from admission to later in the stay), complications (infections, post-op stroke, AMI, etc.), cost, length of stay, and admission to a critical care area. It was not modeled to predict only one specified outcome.

All severity criteria (risk factors) measured by CSI are disease-specific and depend on the ICD-9-CM codes assigned to the patient, the patient's age, and the care setting. The over 2,100 CSI severity criteria are based on objective clinical findings, i.e., the physiological signs and symptoms of a disease, such as temperature, blood pressure, laboratory values, radiology findings, and physical findings. Published examples of CSI disease criteria include sets for adult inpatients with pneumonia, acute myocardial infarction, malignant neoplasm of the large intestine, and injury of lower extremities.¹ The signs and symptoms for adult inpatients with a diagnosis of pneumonia, for example, involve the cardiovascular system, temperature, lab results, x-ray findings, and respiratory findings. All of these signs and symptoms are included in the pneumonia criteria set, and they are presented in four columns according to the extent of abnormality of each. Normal to mild symptoms are in the Level 1 column, moderate symptoms are in Level 2, severe symptoms are in Level 3, and catastrophic or life-threatening symptoms with likelihood of organ failure are in Level 4. Thus, a patient's signs and symptoms for a given disease are represented as a list of scores-1, 2, 3, or 4-depending on the degree of severity of each sign or symptom for that particular patient and his/her age, during a specified time interval and in a specified setting: inpatient, ambulatory, long-term care, etc.

CSI stratifies patients of all ages into homogenous severity of illness groups. There are more than 23,500 separate CSI criteria sets (similar to those referenced above) corresponding to diseases (aggregations of similar ICD-9-CM codes), age groups, and care settings. Note that each of the more than 19,000 ICD-9-CM codes indicates the existence of a disease; it does not indicate the extent (severity) of the disease. The choice of disease-specific, setting-specific, and age-specific severity-leveled criteria to be included in CSI is based on the clinical judgment of expert physicians, the literature, and medical textbooks, not on statistical regression analysis. Thus, the calculation of severity is not influenced by historical data that incorporates possibly inefficient medical practices of the past. CSI criteria are independent of treatments and are not tied to any particular outcome, e.g., mortality. Published work by several researchers^2,3,4,5 has shown CSI severity scores to be among the best predictors of outcomes such as cost, LOS, mortality, resource utilization, and complications.

CSI Data Requirements

There is no fixed list of required data elements for CSI data collection. Because the system is disease-specific, age-specific, and location-specific, the criteria requested for each patient depend on the ICD-9-CM codes assigned to the patient's medical record, the age of the patient, and the care setting. The severity criteria used are based on objective clinical findings and not treatment characteristics, so performance of treatments does not affect the severity level of the patient.

CSI Does Not Depend on Treatments

There are at least two reasons not to include treatment variables in the definition of severity. First, the use of specified treatments may differ because of physician preference or the availability of the treatments within a specified facility. If two patients have identical objective signs, symptoms, and clinical findings for a disease but one receives more treatments, such a greater blood product use or longer intubation time, a treatment-dependent severity system might report that the more extensively treated patient was "sicker". This would be illogical and would render that system useless for identifying true differences in severity.

Second, if a severity system uses treatments to define how sick a patient is, one cannot use that system to control for patient factors in a clinical practice improvement study designed to determine best treatments, since the treatments are included as patient severity criteria. For both these reasons, the CSI software system uses only objective clinical findings, and not treatments, as criteria to define patient severity of illness.

Locations of CSI Data Elements

Data are abstracted from clearly specified locations in the patient medical chart. For example, vital sign information must come from the vital sign flow sheets and not from narrative notes. Lab values must come from laboratory report sheets. Physiologic signs and symptoms may come from physician, nursing, and ancillary services' (respiratory or physical therapy, dietary, etc.) narrative notes.

Computation of CSI Severity Scores

The inputs to the CSI software system's methodology for risk adjustment are the disease-specific, age-specific, and setting-specific severity criteria at specified levels of abnormality in a patient's chart. The intermediate output is a list of severity ratings, an integer between zero and four, one for each of a patient's diagnoses. To form its final output, CSI combines the severity ratings for each separate diagnosis to obtain an overall patient severity level that is presented on a categorical scale of 1 (low severity) to 4 (high severity), as well as on a continuous scale with non-negative integer values that are not subject to any preset maximum limit. Higher numbers mean that patients are more severely ill.

To determine the severity of a specific disease in a specified setting and age group, CSI uses the two most severe criteria; for examples of criteria sets; see reference 4, pp. 116-131. To compute the overall severity score for a patient, CSI determines the severity of each disease using the disease-specific severity factors. It then combines the severity scores for all diagnoses using disease-specific system rules that reflect the interaction of the diagnoses. Thus, to produce the overall severity score, the system's logic takes into account the interactions of the diseases that are present, their severity levels, and the clinical relationships of the diseases, along with the patient's age and setting of care.

Missing data make no contribution to CSI severity scoring. "Points" are not subtracted because specific data elements are missing. All CSI severity scoring is based on data elements found in a patient's chart that are considered out of the normal range for each specific diagnosis listed for the patient. Data completeness reports show completion statistics for each entry field (including severity criteria). If important entry fields are reported to be frequently empty, the ISIS training-team reliability person works with the data collector to promote complete collection of these data.

CSI Improves ICD-9-CM Coding

Although CSI scoring starts with ICD-9-CM codes, the system is not strictly dependent on the coding of the original medical record abstract. This is because the CSI system uses automatic error-checking routines to spot coding errors of two kinds: those in which a diagnosis was assigned in the absence of sufficient evidence in the chart to document that it actually existed or was correct, and those in which a diagnosis was present, but was not assigned. For example, if insufficient data in a chart are found to substantiate the severity criteria associated with a particular diagnosis, that diagnosis receives a severity rating of 0. It is then natural to ask if the diagnosis is correct or if the medical record is fully documented. Similarly, if abnormal signs and symptoms are recorded in the chart but are not prompted by the severity criteria associated with any of the assigned diagnoses, it is natural to ask whether an uncoded condition may be present that should be added to the diagnosis list. For example, if fever, high white blood cell count, and infiltrates in the lungs are recorded in the chart but are not asked for by the software during CSI scoring, perhaps there is a missing diagnosis corresponding to a disease process such a lung infection or pneumonia. Thus, by matching signs and symptoms to diseases, the CSI software system helps validate and improve the ICD-9-CM coding process.

One additional feature of CSI is to designate whether a diagnosis is present on admission or not; hence, one can identify complications, such as infections, that occur after admission and can determine how severe they are by their disease-specific maximum severity score. ISIS's research has found that patients who are sicker on admission are more likely to get infections and other complications during the hospital stay, and also are more likely to have their comorbidities worsen after admission. The CSI software can track all these outcomes. CSI can also measure readmission rates by including a measure of readmission in an Auxiliary Data Module (ADM), or by tracking patients longitudinally over time. Thus, the CSI software system captures a wide variety of outcome measures.

Timing of CSI Severity Reviews

Severity measurements can be performed to determine a patient's severity at any point during the care process, e.g., at multiple times during an inpatient admission or long-term care stay, or during sequential visits to ambulatory providers. CSI coding may be performed either retrospectively or concurrently. Choice of timing of CSI assessments depends upon how the severity data are to be used.

Other Approaches to Measure Severity

Many severity systems (but not the CSI system) are based on statistical regression models constructed to explain the variation in one particular outcome variable, such as in-hospital mortality or cost. Once the coefficients of the model are set, the model is then used to calculate an expected value of the particular outcome variable for any new patient or set of patients. These models are often used to compare the performance of hospitals. Perhaps the best-known example is the Medicare mortality reports that were developed by the US Health Care Financing Administration. Many local community hospital rating projects have also used this methodology to derive "quality" scores for "report cards" on their providers.

A disadvantage of using a severity model derived from statistical regression is that the coefficients reflect the medical practices of the providers included in the modeled data set during a particular period of observation. Yet, who is to say that these providers were exhibiting best practices during that time? Given the enormous variations in practices that have been observed in so many areas of medicine, older data sets are likely to reflect historical inefficiencies and even some suboptimal practices.

Validity of CSI

CSI methodology has continued to be validated. Predictive validity has been tested by how well CSI severity scores predict various outcomes. Although CSI was not designed to explain any particular outcome, e.g., cost, length of stay, or mortality, its scores have been shown to correlate with such severity-dependent outcomes as well or better than those from systems based on explicit models for these outcomes.^2,3,4,5

All CSI criteria sets have been reviewed by panels of medical specialists (physicians, nurses, and other caregivers, e.g., dietitians, respiratory therapists, etc.) for content validity. These experts determine if the criteria are complete and are leveled appropriately to reflect disease-specific severity differences by age and care setting. There is remarkable agreement among clinicians about what criteria should be used and how to specify their severity levels. Criteria sets are re-reviewed periodically to ensure the criteria are current and relevant to changing medical practices. Prior to using the CSI system for severity stratification in health care research or in a Clinical Practice Improvement (CPI) study, providers are encouraged to review the CSI criteria sets and recommend refinements to them.

Construct validity is much harder to measure. Constructs represent efforts to measure relatively abstract variables, which is the case with severity. Constructs vary widely in the extent to which the domain of related observable variables is (a) large or small and (b) tightly or loosely defined. For severity measurement, the domain of variables is vast. It is difficult to define which variables do or do not belong in the domain and the boundaries of the domain or related observables are not clear. There are three major aspects in the process of validating a construct: (1) specifying the domain of observables, (2) determining to what extent all or some of those observables correlate with each other or are affected alike by treatments, and (3) determining whether or not one, some, or all measures of such variables act as though they measure the construct.

In developing CSI, the domain of observables was specified as follows: first, the design team decided that all of the patient's diseases and their respective severity levels had to be taken into account. The design team developed a mechanism to account for levels of each criterion specific to the corresponding diseases (ICD-9-CM codes), a patient's age, and care setting, and a mechanism to account for the interactions of all of a patient's diseases, so that an overall severity score could be created. After much empirical research, we decided that the two most severe criteria would determine a disease-specific severity level. Second, unlike many other systems, CSI was not developed to predict any single outcome based on a specified derivation set. Some severity systems have used physiologic criteria (e.g., PRISM, APACHE, MediQual), some have used diagnoses (e.g., DRGs), and some have tried to combine information from all diagnoses present (e.g., APR-DRGs). Only the CSI system has put all the observables together in one system. CSI has clearly specified the domain of observables.

Now consider the second aspect of construct validity. As patients get sicker, more of their body systems show signs of failure. CSI picks up those multiple signs and symptoms for each disease, and, thus, correlates with the greater difficulty of bringing a sicker patient back to baseline health. However, until there is a well-specified domain for measuring severity of illness, there is no way to know exactly which studies should be done to test the adequacy with which a construct is measured. Strictly speaking, scientists can never be sure that a construct has been measured or that a theory regarding that construct has been tested, even though it may be useful to speak as though such were the case.⁶

Reliability of CSI

Reliability testing monitors the accuracy of data collection. The standard method employed by ISIS to ensure inter-rater reliability is as follows: the ISIS training-team reliability person selects several of the first 10 charts abstracted for CSI data by each data collector and abstracts independently each of these charts. Results from the ISIS abstraction are then compared with those from the data collector's abstraction. A high agreement rate (at least 95% percent) at the criteria level between each data collector and the ISIS reliability trainer is expected. If a data collector fails to achieve at least this rate, the ISIS reliability trainer works with the individual to improve abstraction methods and accuracy. One or more subsequent reliability tests are conducted for each data collector until an agreement rate of at least 95% is achieved. Reliability testing is conducted again after one month of data collection has been completed. The reliability trainer randomly selects three patient records from data submitted by each data collector. The data collector is then asked to photocopy, blind for confidentiality, and submit to ISIS the corresponding patient medical records. The same minimum high agreement rates are expected. Reliability testing is then conducted every six months throughout the data collection period. Reliability testing at the beginning of data collection is intended to ensure that data collectors have learned correct abstraction methods and understand the definition of all variables. Continuing reliability testing ensures that data collection continues to be accurate.

Predictability of CSI

Thomas and Ashcraft² used the R² statistic to measure how well various severity measures explained variations in estimated costs among patients hospitalized for each of eleven conditions. Six severity methods were evaluated: AIM (Acuity Index Method), APACHE II, the CSI admission and maximum scores, Disease Staging, the MedisGroups admission and morbidity ratings, and PMC (Patient Management Categories). Analyses were performed twice, once on all cases and once with outliers eliminated. Two forms of models were considered: first, assuming simple linear relationships between severity and costs (therefore entering a single, continuous variable into the model); and second, allowing for nonlinear relationships (modeled with categorical risk variables). Modeling was performed separately within each of eleven conditions.

Table 1 shows the R² values from the nonlinear models used to predict cost, averaged across the eleven conditions from the Thomas and Ashcraft study. The performance of the severity measures varied depending on whether outliers were included in the analysis and whether R² was calculated from the model fit to the development data set, or whether a validation R² was estimated. The CSI software system earned the highest R² when all cases were analyzed as well as when outliers were eliminated. Thus, CSI was the best predictor of cost in this study.

Using a sample of approximately 76,000 cases abstracted from 25 hospitals in New Jersey, Averill and colleagues³ examined the relationship between maximum CSI severity scores and hospital costs. Results were reported for 76 DRGs, selected based on preliminary analyses that identified DRGs in which severity affected patient costs. Only inlier patients (i.e., those without extreme outcome values) were analyzed. For 34 DRGs, the R² for the CSI severity scores within the DRGs was between 0.10 and 0.19; for 25 DRGs, it was between 0.20 and 0.29; and for 17 DRGs, R² was greater than or equal to 0.30. Thus, CSI explained much of the variation within DRGs.

In summary, the CSI system behaves as expected: sicker patients have longer lengths of stay, are more costly, have higher death rates, have more complications, and have more visits. CSI takes into consideration all relevant patient signs and symptoms for every disease.

Table 1: R² For Predicting Cost by Risk-Adjustment System