- Research article
- Open Access
- Open Peer Review
This article has Open Peer Review reports available.
Development and validation of a bedside risk score for MRSA among patients hospitalized with complicated skin and skin structure infections
© Zilberberg et al.; licensee BioMed Central Ltd. 2012
Received: 17 January 2012
Accepted: 25 June 2012
Published: 11 July 2012
Methicillin-resistant Staphylococcus aureus (MRSA) is a frequent cause of complicated skin and skin structure infections (cSSSI). Patients with MRSA require different empiric treatment than those with non-MRSA infections, yet no accurate tools exist to aid in stratifying the risk for a MRSA cSSSI. We sought to develop a simple bedside decision rule to tailor empiric coverage more accurately.
We conducted a large multicenter (N=62 hospitals) retrospective cohort study in a US-based database between April 2005 and March 2009. All adult initial admissions with ICD-9-CM codes specific to cSSSI were included. Patients admitted with MRSA vs. non-MRSA were compared with regard to baseline demographic, clinical and hospital characteristics. We developed and validated a model to predict the risk of MRSA, and compared its performance via sensitivity, specificity and other classification statistics to the healthcare-associated (HCA) infection risk factors.
Of the 7,183 patients with cSSSI, 2,387 (33.2%) had MRSA. Factors discriminating MRSA from non-MRSA were age, African-American race, no evidence of diabetes mellitus, cancer or renal dysfunction, and prior history of cardiac dysrhythmia. The score ranging from 0 to 8 points exhibited a consistent dose–response relationship. A MRSA score of 5 or higher was superior to the HCA classification in all characteristics, while that of 4 or higher was superior on all metrics except specificity.
MRSA is present in 1/3 of all hospitalized cSSSI. A simple bedside risk score can help discriminate the risk for MRSA vs. other pathogens with improved accuracy compared to the HCA definition.
Although rates of serious infections with pathogens such as methicillin-resistant Staphylococcus aureus (MRSA) have been rising over recent years , there is encouraging news from the Centers for Disease Control and Prevention (CDC) indicating that this rise may be abating for healthcare-associated pathogens, though not for community-acquired ones . Much of the emergent resistance can be attributed to selection pressures created by overuse of broad-spectrum antibiotics [3, 4]. Empiric treatment of an infection becomes a balancing act for physicians concerned with containing the spread of resistance vs. not covering broadly enough, impacting clinical response and potentially outcomes [5–11].
Complicated skin and skin structure infections (cSSSI) are a common reason for hospitalization. S. aureus-related cellulitis hospitalizations, for example, have risen 4-fold in a recent 6-year period to over 90,000 discharges , and MRSA is the cause of cellulitis in 15% of all cSSSI cases . While there is indirect evidence that inappropriate choice of empiric therapy may impact clinical outcomes among patients hospitalized with cSSSIs , this has not been confirmed broadly . However, what does appear to be affected by the initial antibiotic choice is the duration of hospitalization , an outcome of utmost importance given the fiscal constraints in the healthcare system. For this reason, an early risk stratification tool for the presence of MRSA infection would be useful because a) prompt targeted treatment would balance appropriate spectrum coverage with concerns for resistance emergence and b) conventional laboratory testing takes 48–72 h to yield culture results.
We set out to compare current epidemiology and outcomes of patients hospitalized with MRSA cSSSI to those hospitalized with non-MRSA. Based on these differences, we developed and validated a clinical decision tool to help guide appropriate antimicrobial treatment at presentation.
This retrospective cohort study used data collected from 62 hospitals in the Health Facts electronic medical record database (Cerner Corporation, Kansas City, MO). Health Facts is a database built from hospitals’ comprehensive clinical records including pharmacy, laboratory, admission, and billing information from all affiliated patient care locations. All admissions, medication orders, and laboratory orders and collections are labeled with date and time. Hospital billing/registration and encounter data are integrated with clinical information relating to the drugs dispensed and the results of diagnostic laboratory testing. Cerner Corporation has established Health Insurance Portability and Accountability Act (HIPAA)–compliant operating policies and procedures using statistical methods to establish de-identification.
Included patients were at least 18 years of age, hospitalized between April 2005 and March 2009, and met our diagnostic criteria in one of two ways: 1) Admission had a primary diagnosis from Additional file 1: Appendix A OR 2) Admission had a secondary diagnosis from Additional file 1: Appendix A, a primary diagnosis of sepsis (International Classification of Diseases, Clinical Modification [ICD-9-CM] codes 020.2, 038.xx, 790.7, 995.91, or 995.92), and no other source of infection documented by ICD-9-CM code. Patients were also required to have both 1) collection of a blood or skin culture that was positive for bacteria and 2) treatment with any intravenous (IV) antibiotic within 48 h of presentation. The positive index culture had to contain at least one organism that was not considered to be a skin contaminant. Cultures of organisms considered to be skin contaminants (Corynebacterium/diphtheroids, Staphylococcus epidermidis or other coagulase-negative Staphylococcus sp., Propionibacterium, Streptococcus viridians, Aerococcus spp, Bacillis spp [except B. anthracis]) were excluded from the cohort. Other exclusions were pregnancy or complications of childbirth, major trauma, and inpatient admission within 30 days with primary or secondary diagnosis of cSSSI (Additional file 1: Appendix A).
Microbiology results were prioritized to reduce the likelihood of colonizing microorganisms, though included patients could have polymicrobial infection. For patients with multiple culture results within the 48-hour timeframe, cultures from a skin source or venipuncture were prioritized over blood cultures drawn from a line, and certain sources of skin cultures with a high risk of contamination such as “swabs” were excluded. Lower extremity infections were identified using the 4th digit of the ICD-9-CM diagnosis code and/or the anatomic location of the skin culture, if present. Healthcare-associated (HCA) infection was defined based on the presence of at least one of the following factors: 1) admission from a chronic care facility, or 2) hospitalization within the previous 180 days or recent outpatient surgery, or 3) hemodialysis within 90 days or end-stage renal disease (ESRD) diagnosis code, or 4) chronic dependence on mechanical ventilation. Impaired immune function, based on specific ICD-9-CM codes and/or medication use, was examined separately. Chronic comorbidities present during admission or 12 months prior were identified by ICD-9-CM codes; baseline (within 48 hours of admission) laboratory variables were also recorded. Organ dysfunction measures at or before the time of the index culture were designed to be consistent with a Sepsis-related Organ Failure Assessment (SOFA) score equal to 2 or greater . These included cardiovascular, respiratory, hepatic, hematologic, and renal dysfunction. Unadjusted hospital mortality served as the primary endpoint, while length of stay (LOS) among survivors was the secondary outcome.
Based on statistical and clinical considerations, we built and explored the usefulness of a simple summary score to identify infection with MRSA vs. a non-MRSA organism.
Our aim was to develop a simple summary score to be used at the bedside to serve as a clinical decision tool to distinguish patients with MRSA from those with an infection with another pathogen (non-MRSA). To accomplish this, we first compared patients admitted with MRSA to those with non-MRSA with regard to baseline demographic, clinical and hospital characteristics. To test for differences between groups, we used t-tests for continuous variables and the chi square test or Fisher’s exact test for categorical variables. When constructing the summary scores, potential covariates were included in the models if they reached significance at alpha=0.2 in the univariate analyses, or if a priori they were determined to have clinical relevance (Additional file 1: Appendix B) . Variables were included only if they were present on admission or could be known within the first 24 hours of admission. The rationale for the 24-hour window was that inappropriately targeted empiric antimicrobial coverage within this time frame is associated with worsened outcomes [5–11]. In the selection of laboratory variables, we only considered tests for which at least 95% of values were obtained within 24 hours of admission.
A split sample approach was used, whereby 80% of the patients were randomly selected for model development and the remaining 20% were used for model validation. To identify the most predictive variables of MRSA vs. non-MRSA infection, we constructed a series of backward stepwise logistic regression models on bootstrapped with replacement (simulated) datasets from the development set and created a frequency table indicating the number of times the eligible predictors entered these stepwise regression models. The variables appearing most frequently were then considered for inclusion in the final summary score . The score was designed a priori to rely on no more than 5 predictors for ease of use in clinical practice (which compelled us to exclude highly predictive but rare variables from the final model). The final model’s predictors were included in a logistic regression model and the coefficients were fully standardized to a unit variance. The predictors in the summary score were then assigned integer-valued weights proportional to these standardized coefficients.
We calculated the prevalence of MRSA infection at every score value in the development cohort and compared these results to the prevalence in the validation cohort to assess calibration. A simple, overall classification rate is often misleading when the overall prevalence of a condition is either quite high or low. To better assess the predictive ability of the models, we calculated specificities and sensitivities at various threshold levels of the score(s) to derive likelihood ratios. With these statistics, we constructed Fagan’s nomograms to show a patient’s posterior probability of having MRSA if the score was above or below a particular threshold (i.e., being above a threshold level was analogous to a positive “test”) .
To augment usefulness at the bedside and to benchmark the score against the HCA risk factors, an additional assessment of score characteristics was done calculating sensitivity, specificity, positive and negative predictive values for each threshold score along the scale. These characteristics were then compared to the performance of the HCA definition as a way to discriminate the risk for MRSA from that for non-MRSA infection.
Patient demographics and hospital characteristics
Total sample (N= 7183)
MRSA (N = 2387)
Non- MRSA (N = 4796)
Age in years, mean (SD)
Year of discharge
Urban setting (vs. rural)
Hospital bed size
Teaching (vs. non-teaching)
Charlson comorbidity index, mean (SD)
Risk factors for healthcare-associated infection (any)
Recent hospital discharge (within 180 days) or recent outpatient surgery
Outpatient hemodialysis within 90 days or ESRD patient
Admission from a chronic care facility
Mechanical ventilation/Dependence on respirator
Immunosuppression Factors (any below), N (%)
ESRD (current or prior)
Received systemic corticosteroid/immunosuppressive/chemotherapy
Leukemia, lymphoma, metastasis
Aplastic anemia and pancytopenia
Other immunosuppression factors**
Comorbidity (current or 12 months prior to admission unless noted), N (%)
Diabetes mellitus by diagnosis code or medications
Diabetic lower extremity infection (current encounter)
Renal dysfunction at baseline
Coronary artery disease
Complication of electronic internal device (ICD-9 Code = 996.6, current encounter)
Chronic obstructive pulmonary disease
Cardiac dysrhythmias (prior to index encounter)
Acute coronary syndrome/angina
Hematologic dysfunction at baseline
Cirrhosis/chronic liver disease
Laboratory values, mean (SD)
White blood cell count (k/mm3)
Serum creatinine (mg/dL)
Platelet count (k/mm3)
Unadjusted hospital mortality was lower in the MRSA (3.0%) than the non-MRSA (4.7%) group, p<0.001. Similarly, the median hospital LOS was shorter in the MRSA (4.2 days) than in the non-MRSA (5.7 days) group, p <0.001.
MRSA prediction score
MRSA risk prediction score model
Skin Infection, any site
Sub Score =
Black Race; If true add 1
No Evidence of Diabetes Mellitus (DM) or Presence of Medication for the Treatment of DM; If true, add 1
No Evidence of Cancer (eg, lymphoma, leukemia):
If true, add 1
No Evidence of Renal Dysfunction:
No known outpatient Hemodialysis within 90 Days of Admission
known ESRD patient
current Serum Creatinine > 2 mg/dL; if true, add 1
Presence of Cardiac Dysrhythmias Prior to Index Encounter;
If present (if true); add 1
Prevalence of MRSA at different levels of the MRSA score in the dataset
When the MRSA Score
0 or 1
7 or 8
Percent of Observed
Patients with MRSA
Test characteristics for various thresholds of the MRSA score and the HCA risk factors on the validation set
MRSA score (0 to 8)
Positive predictive value
Negative predictive value
Overall prevalence of patients with the score value or higher in the validation set
7 or 8
HCA risk factors
In this large multi-center cohort study, we have demonstrated that MRSA is present in one-third of all cSSSI patients hospitalized for their infection. There was a distinct age difference with regard to the risk for MRSA, wherein younger age groups were more likely to be infected with it relative to the older ones. Similarly, the burdens of comorbidities and risk factors for healthcare-associated infection were lower in the MRSA-infected group compared to those harboring non-MRSA. Commensurately, both hospital mortality and LOS among the MRSA-infected group were lower when compared to the non-MRSA group. Finally, we developed a bedside risk score to aid in the early identification of MRSA as a potential pathogen. Although improving somewhat the post-test probability of MRSA, the usefulness of the score may be limited by the high pathogen prevalence in the cohort.
Over the last decade, the increasing frequency of resistant organisms among persons presenting with infections from the community prompted a re-evaluation of the populations at risk for harboring such pathogens. Out of this effort, the HCA infection definition was born to signify a patient presenting from the community who is nevertheless put at risk for resistant pathogens by his/her ongoing contact with the healthcare system . Some of the commonly acknowledged risk factors for HCA infection are recent hospitalization, immune suppression, need for hemodialysis and nursing home residence. Because the HCA risk factors were neither developed nor validated prospectively, several authors have expressed concern that such imprecise definition would lead to over-diagnosis of resistance, and thus overtreatment, with its sequelae of promoting further resistance development. For example, in a cohort of patients with HCA pneumonia (HCAP), Shorr et al. calculated the specificity of the HCAP risk factors for a resistant pathogen to be below 50% . In an effort to sharpen accuracy, the investigators focused on identifying specific risk factors for resistant pathogens overall and for MRSA individually, with the view to developing a simple bedside risk score [17, 18].
Our findings are important on several accounts. The risk factors we found that were strongly associated with the presence of a MRSA infection differed from the broadly used HCA definition. In fact, HCA was less likely to be present in the setting of MRSA than non-MRSA infection. Although this differs from other reports that found 2/3 of all MRSA cSSSIs to have underlying HCA risk factors, the discrepancy is likely due more to the lower prevalence of HCA overall in the current study rather than to fundamental microbiologic differences . That is the prior study noted ~75% prevalence of HCA in the cohort of 717 patients admitted with a cSSSI, while in the current analysis it was only 35%. Given that the previous result came from a single-center analysis, the current data are more generalizable. Additionally, although the systematically derived score performed somewhat better than the consensus definition, it was far from perfect, thus shielding some, but not all, of those with susceptible pathogens from exposure to overly broad coverage. A similar finding was reported by Schreiber et al. in a group of patients with pneumonia and respiratory failure .
In the realm of cSSSI, the evidence-based guideline recommendations shed limited light on how to evaluate a patient’s risk for harboring MRSA . Compounding this difficulty further is the emergence of distinct community-acquired genotypes: though they require expanded antimicrobial coverage, their risk factors diverge from the traditional HCA paradigm. Although we were unable to examine the genetic make-up of the organisms in our study, the finding that MRSA was more frequent in the younger age groups may be consistent with the high prevalence of CA-MRSA. This underscores the need for a better risk stratification tool in this area than the HCA definition alone can provide.
Our current effort extends the findings of previous investigators on the challenges of identifying resistant organisms upon presentation and prior to the availability of culture results. In our study, the HCA definition identified only 32.0% of patients with MRSA correctly, while mislabeling an additional 35.5% as having MRSA, thus potentially prompting a high volume of overuse of broad-spectrum coverage (Table 4). In contrast, different values for our MRSA score provide flexibility for balancing sensitivity with specificity, and, therefore the negative with the positive predictive values. Namely, if excluding the possibility of MRSA is of primary interest, then the MRSA score of 2 or higher provides a higher degree of confidence than the HCA definition, given its substantially higher negative predictive value. However, when accuracy in general is sought, it is clear that higher MRSA scores (e.g., 4 or more) possess better test characteristics than the HCA definition, thus leading to more targeted prescription and less potential misuse of antibiotics.
Our study has a number of strengths and limitations. As a large multicenter cohort representing over 60 US hospitals, it has a high degree of generalizability. At the same time, as a retrospective observational study it is subject to a number of threats to validity, including a selection bias. We mitigated the possibility of this by setting a priori cohort definitions and including all consecutive patients meeting our selection criteria. Given that we relied on administrative coding to define our cohort, the study is prone to some degree of misclassification. However, the same approach has been used widely by multiple investigators, and this type of misclassification is likely to be either minor or non-differential [11, 12, 22]. Finally, while we focused on predictors that had a strong association with MRSA and were common, it is possible that other variables not in our database would have yielded a more accurate model, though we note that the list of predictors examined was quite comprehensive.
In summary, although our simple bedside score did improve on the specificity of the HCA definition, it left room for further improvement. Ultimately, what is needed is a bedside diagnostic tool that can identify MRSA and other resistant organisms rapidly and with a high degree of accuracy, so as to tailor treatment appropriately. Until such tools are available, and depending on the local patterns of resistance, empiric broad-spectrum coverage followed by prompt de-escalation in response to microbiology data may need to remain the standard of care.
This study was funded by Astellas Pharma US (APUS), Inc, Deerfield, IL. Drs. Zilberberg and Shorr have received research and consulting funds from APUS. Dr. Chaudhari is an employee and a stock holder in APUS. Drs. Campbell and Emons, Ms. Fiske and Mr. Hays work for Cerner Corporation, which has received research funding from APUS. Dr. Nathanson works for OptiStatim, LLC, which has received funds from Cerner Corporation.
- Klein E, Smith DL, Laxminarayan R: Hospitalizations and deaths caused by methicillin-resistant Staphylococcus aureus, United States, 1999–2005. Emerg Infect Dis. 2007, 13 (12): 1840-1846. 10.3201/eid1312.070629.View ArticlePubMedPubMed CentralGoogle Scholar
- Kallen AJ, Mu Y, Bulens S, Reingold A, Petit S, Gershman K, et al: Health care-associated invasive MRSA infections, 2005–2008. JAMA. 2010, 304 (6): 641-648. 10.1001/jama.2010.1115.View ArticlePubMedGoogle Scholar
- Dellit TH, Owens RC, McGowan JE, Gerding DN, Weinstein RA, Burke JP, et al: Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America guidelines for developing an institutional program to enhance antimicrobial stewardship. Clin Infect Dis. 2007, 44 (2): 159-177. 10.1086/510393.View ArticlePubMedGoogle Scholar
- Spellberg B, Guidos R, Gilbert D, Bradley J, Boucher HW, Scheld WM, et al: The epidemic of antibiotic-resistant infections: a call to action for the medical community from the Infectious Diseases Society of America. Clin Infect Dis. 2008, 46 (2): 155-164. 10.1086/524891.View ArticlePubMedGoogle Scholar
- Micek ST, Kollef KE, Reichley RM, Roubinian N, Kollef MH: Health care-associated pneumonia and community-acquired pneumonia: a single-center experience. Antimicrob Agents Chemother. 2007, 51 (10): 3568-3573. 10.1128/AAC.00851-07.View ArticlePubMedPubMed CentralGoogle Scholar
- Schramm GE, Johnson JA, Doherty JA, Micek ST, Kollef MH: Methicillin-resistant Staphylococcus aureus sterile-site infection: The importance of appropriate initial antimicrobial treatment. Crit Care Med. 2006, 34 (8): 2069-2074. 10.1097/01.CCM.0000227655.41566.3E.View ArticlePubMedGoogle Scholar
- Ibrahim EH, Sherman G, Ward S, Fraser VJ, Kollef MH: The influence of inadequate antimicrobial treatment of bloodstream infections on patient outcomes in the ICU setting. Chest. 2000, 118 (1): 146-155. 10.1378/chest.118.1.146.View ArticlePubMedGoogle Scholar
- Alvarez-Lerma F: Modification of empiric antibiotic treatment in patients with pneumonia acquired in the intensive care unit. ICU-Acquired Pneumonia Study Group. Intensive Care Med. 1996, 22 (5): 387-394.Google Scholar
- Iregui M, Ward S, Sherman G, Fraser VJ, Kollef MH: Clinical importance of delays in the initiation of appropriate antibiotic treatment for ventilator-associated pneumonia. Chest. 2002, 122 (1): 262-268. 10.1378/chest.122.1.262.View ArticlePubMedGoogle Scholar
- Zilberberg MD, Shorr AF, Micek ST, Mody SH, Kollef MH: Antimicrobial therapy escalation and hospital mortality among patients with health-care-associated pneumonia: a single-center experience. Chest. 2008, 134 (5): 963-968. 10.1378/chest.08-0842.View ArticlePubMedGoogle Scholar
- Zilberberg MD, Shorr AF, Micek ST, Chen J, Ramsey AM, Hoban AP, et al: Hospitalizations with healthcare-associated complicated skin and skin structure infections: impact of inappropriate empiric therapy on outcomes. J Hosp Med. 2010, 5 (9): 535-540. 10.1002/jhm.713.View ArticlePubMedGoogle Scholar
- Lipsky BA, Weigelt JA, Gupta V, Killian A, Peng MM: Skin, soft tissue, bone, and joint infections in hospitalized patients: epidemiology and microbiological, clinical, and economic outcomes. Infect Control Hosp Epidemiol. 2007, 28 (11): 1290-1298. 10.1086/520743.View ArticlePubMedGoogle Scholar
- Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine, et al: The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Intensive Care Med. 1996, 22 (7): 707-710. 10.1007/BF01709751.View ArticlePubMedGoogle Scholar
- Austin PC, Tu J: Bootstrap methods for developing predictive models. Am Stat. 2004, 58: 131-137. 10.1198/0003130043277.View ArticleGoogle Scholar
- Guidelines for the management of adults with hospital-acquired ventilator-associated, and healthcare-associated pneumonia. Am J Respir Crit Care Med. 2005, 171 (4): 388-416.Google Scholar
- Fagan TJ: Letter: Nomogram for Bayes theorem. N Engl J Med. 1975, 293 (5): 257-PubMedGoogle Scholar
- Shorr AF, Zilberberg MD, Micek ST, Kollef MH: Prediction of infection due to antibiotic-resistant bacteria by select risk factors for health care-associated pneumonia. Arch Intern Med. 2008, 168 (20): 2205-2210. 10.1001/archinte.168.20.2205.View ArticlePubMedGoogle Scholar
- Shorr AF, Zilberberg MD, Reichley R, Kan J, Hoban A, Hoffman J, et al: Validation of a clinical score for assessing the risk of resistant pathogens in patients with pneumonia presenting to the emergency department. Clin Infect Dis. 2012, 54 (2): 193-198. 10.1093/cid/cir813.View ArticlePubMedGoogle Scholar
- Zilberberg MD, Shorr AF, Micek ST, et al: Epidemiology and outcomes of hospitalizations with complicated skin and skin structure infections: Implications of healthcare associated infection risk factors. Infect Control Hosp Epidemiol. 2009, 30: 1203-10. 10.1086/648083.View ArticlePubMedGoogle Scholar
- Schreiber MP, Chan CM, Shorr AF: Resistant pathogens in nonnosocomial pneumonia and respiratory failure: is it time to refine the definition of health-care-associated pneumonia?. Chest. 2010, 137 (6): 1283-1288. 10.1378/chest.09-2434.View ArticlePubMedGoogle Scholar
- Stevens DL, Bisno AL, Chambers HF, Everett ED, Dellinger P, Goldstein EJ, et al: Practice guidelines for the diagnosis and management of skin and soft-tissue infections. Clin Infect Dis. 2005, 41 (10): 1373-1406. 10.1086/497143.View ArticlePubMedGoogle Scholar
- Edelsberg J, Berger A, Weber DJ, Mallick R, Kuznik A, Oster G: Clinical and economic consequences of failure of initial antibiotic therapy for hospitalized patients with complicated skin and skin-structure infections. Infect Control Hosp Epidemiol. 2008, 29 (2): 160-169. 10.1086/526444.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2334/12/154/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.