We describe the derivation and internal validation of a scoring system able to predict the probability of nasal MRSA carriage and the expected PPV of a molecular screening test in admissions to a European tertiary referral center with low prevalence. In the assessment of admitted patients published models mainly focus on measures of discrimination, which summarize the ability to separate true positive from negative results. Many models assessed for their discriminatory ability have been created with the primary aim of restricting screening efforts to a subset of hospital admissions. Nevertheless, it seems questionable, if predictive models can attain high degrees of separation into carriers and non-carriers at all, as even elaborate models fail to display acceptable discrimination and thus sensitivity in detecting unknown carriers when screening is restricted to a subset of the admitted population. E.g. a recently published model with comparably good discriminatory ability (AUC 0.72 in ROC analysis) would have missed 32% of carriers even if 30% of admissions had been screened
. This seems to be an inherent limitation of prediction tools, since attainable AUCs depend on the distribution of risk in the population under study; specifically, only rarely encountered distributions displaying unusually wide spread, such as the uniform or u-formed distributions, allow AUCs (and with it discriminatory ability) to surpass values of 0.80
. Given this imperfection, it seems futile to continue to design prediction tools for the sole purpose of suggesting who to screen on admission.
In addition to discrimination, however, models can be used for accurate prediction of disease probability, which is of primary clinical interest for the management or prevention of disease
. As an example, the European guidelines on cardiovascular disease prevention stress the importance of absolute risk for the planning of preventive measures
. Similarly, decisions regarding treatment, e.g. the choice of antibiotic before availability of cultural results in the presence of clinical infection, or infection control, e.g. choice of screening method or preliminary measures such as isolation, could be informed by predicted probability of MRSA carriage.
Using a set of potentially readily available predictors we established a predictive model and transformed its output into a score ranging from 0 to 200, whereby predictors were assigned point values in keeping with the sizes of the adjusted coefficients derived from the logistic regression model (Table
2). Unlike published models including factors such as previous antibiotic consumption
[11–14, 26, 27], we have restricted predictors to a set of unequivocal variables potentially extractable from the hospital information system on the day of admission. The advantage of this strategy is that risk assessment would not depend on additional clinician input, who may have competing priorities, particularly in busy services
. Several authors identified similar risk factors to the ones in our model including age
[11, 14, 29–33], nursing home residence
[11, 14, 34, 35], and emergency admission
[30, 36]. Comparable to the group of admission diagnoses under the ICD-10-GM headings “A” (“Certain infections and parasitic diseases”) or “J” (“Diseases of the respiratory system”), other authors found certain infections including pneumonia and wound infections
, skin and soft tissue infections
, skin and bone infection
, and lung disease
 present at admission significantly associated with MRSA carrier status. Nevertheless, several groups failed to find an association between risk factors identified in this study and MRSA carriage; e.g. Sax et al. did not find a significant association to nursing home residence or age > 85 years
, while Harbarth et al. reported no association to admission through the emergency department
. It is difficult to interpret the high prevalence in the medical emergency department within DC compared to the rest of the wards; while it refers patients to other wards, more than 96% of patients in DC and IVD did not pass through the medical emergency department. Therefore it seems unlikely, that it filtered MRSA carriers out before they arrived to other wards. In contrast, the particularly low rate in Anesthesiology within DC (0 of 162, Table
1; 95% CI 0.0% to 2.3% according to binomial distribution) could have been due to chance, as prevalence in this unit within IVC (1.6%) did not differ significantly. Further, we were unable to confirm an independent influence of sex reported by several groups
[11, 14]. These differences likely reflect the inclusion of comparably weak or setting-specific risk factors; e.g. the fact that, in the present study, medical emergency admission had the lowest odds ratio of 1.9 among categorical variables tested and a comparably high p-value of 0.03 (Table
2) may explain why it is rarely reported as independent predictor in other studies.
Our score showed a useful fit within the DC as shown graphically (Figure
2) and using the Hosmer-Lemeshow test on 10 strata. We validated our score on a separate cohort (IVC) admitted to wards F, K, L, and M between August 2008 and January 2012. The prevalence in IVC compared to DC was lower with 1.7% vs. 2.3%, respectively. The reason for the differing prevalence was mainly due to lower previous prevalence within wards F, K, L, and N within DC (1.9%, Table
1). The model’s calibration in IVC was acceptable as shown by the Hosmer-Lemeshow-Test on 10 strata, suggesting that the small difference in prevalence did not necessarily require correction of the score. Nevertheless, correction resulted in more accurate fit, evidenced by the lower χ2- value of the Hosmer-Lemeshow-Test (Figure
6). Furthermore, while both corrected and uncorrected scores enabled classification of individuals into strata of increasing carriage probability, categorization using corrected scores more accurately reflected actual probability (Table
3). The proposed correction therefore facilitates use of our models in settings with different prevalence and thus promotes external validation. While the presented score correction represents a novel application in the area of MRSA prediction, its concept is not new: e.g. adjustment to background rates in cardiovascular risk are commonly applied to achieve accurate prediction, as exemplified by the recalibrated cardiovascular risk chart for Germany
Apart from its effect on calibration, prevalence differences greatly affect the PPV of rapid diagnostic tests. Similar to other reports
[16–18] we confirm that a commercial molecular test based on the amplification of a fragment spanning SCCmec and orfX has low PPVs in a low prevalence setting. The substantial proportions (> 66%) of methicillin-susceptible S. aureus strains lacking mecA among false positives contributed to this phenomenon. Although the specificities of the test were almost equal with 98.7% and 98.6% for DC and IVC, respectively, a small difference in prevalence of 0.6% caused a sizeable drop of PPV from 64.8% (70 of 108) to 54.0% (34 of 63). As higher prevalence is associated with higher PPV of molecular tests
, we explored whether individual probability of carriage showed a similar relationship. Using non-linear regression, we fitted a function corresponding to the mathematical description of PPV by specificity, sensitivity, and individual odds (represented by the score) to observed PPV in eight strata with different scores (Figure
3). According to the Hosmer-Lemeshow-Test on a subset of DC with positive molecular test results, the fit was found acceptable. We then validated this model on IVC using the Hosmer-Lemeshow-Test on five strata. The reduction of strata compared to DC was due to the reduced number of positive results in IVC (in keeping with lower prevalence and sample size), as we aimed for a comparable number of positive results in each stratum; for DC and IVC each stratum thus had an average of 13.5 (108/8) and 12.6 (63/5) positive results, respectively. The fit was found useful before correction, yet in analogy to prediction of probability, it was improved after adjustment for prevalence. This was also evident from the reclassification tables, which demonstrated a more accurate categorization into pre-specified strata. Hence, in addition to the prediction of probability of carriage, our model also predicted the expected PPV of a rapid test for a given individual.
The score can be derived in external settings from a set of patient parameters (containing information on age, nursing home residence, emergency admission, and admission diagnosis) in two steps: 1) an initial score is calculated by substracting 10 from the age and further adding scores according to column “Score” in Table
2; 2) if external MRSA prevalence differs from 2.3% (prevalence of DC), a correction score, derivable from the nomogram in Figure
5 according to prevalence, has to be added to the initial score. The resulting final score can then be translated into probability of carriage and expected PPV using the nomogram in Figure
4. Probability of carriage can inform several clinical decisions. Empiric treatment in case of e.g. wound infections could include antibiotics suitable against MRSA if high probability of carriage is likely (e.g. ≥ 6%); also, preemptive isolation or further control measures could be automatically carried out in these cases. Further, the prediction of PPV could help target molecular tests to individuals where equivocal results are unlikely. Clearly, the formulation of a PPV-threshold above which molecular tests should be applied is a matter of taste; we propose that rapid diagnostic testing be restricted to individuals with expected PPV over 80%, and that cultural screening be used for the rest of admissions. For IVC, this threshold would include only 4.0% of individuals.
Our study has several limitations. In contrast to reports with screening compliance exceeding 80%
[11, 13, 14, 29, 30], only 61% of eligible patients had been screened correctly in our derivation cohort. This was due to the high number of patients that had to be excluded due to testing of sites other than the nose and tests taken more than 2 days after admission. While incomplete coverage of patients may have introduced a bias, the models generated from DC still performed favorably in a separate cohort (IVC). Also, the internal validation cohort consisted of a convenience sample retrospectively extracted from a subset of wards that coincidentally pilot-tested an electronic nursing history system during different periods after August 2008. The different sampling method may have decreased comparability between DC and IVC. Specifically, it remains unknown, why age was different between the two cohorts. Increased age in IVC may have been due to demographic changes within the admitted patients, as the proportion of males remained the same (even when compared to wards F, K, L, M in DC), and increase of age was not associated with increased prevalence, as might have been expected in the case of selective inclusion of older patients. The use of ICD-10-GM admission diagnoses might also have introduced a bias, since the degree of inter-rater agreement in coding admission diagnoses is not well established. Nevertheless, it is conceivable that agreement on ICD-10 chapter headings, as used in our study, will be reasonably high even between raters in different countries. Also, availability of coded admission diagnoses within the first day of admission may pose a hindrance for the use of our models: in our center, only 49% and 66% of admission diagnoses were available in IVC as ICD-10-GM codes on the day of admission and one day after admission, respectively; thus, speedy coding of admission diagnoses (alternatively of chapter headings) has to be implemented for the automated use of our models. Nevertheless, as the most authorative classification of diseases, the choice of ICD-10-GM seems natural; further research could elucidate, whether codes including U80.0 (S. aureus with methicillin or other resistance), Z22.3 (carrier of staphylococci or other bacteria), or Z29.0 (isolation) predicts MRSA carriage. While neither diagnosis was coded as admission diagnosis in our cohorts, the previous use of these codes as secondary diagnoses may well be predictive; nevertheless, this would require that coding data from other hospitals be made available to the admitting center. Finally, our sample size was rather small for a low prevalence area with 3091 and 2043 individuals, respectively; this likely decreased our power to find weak associations between predictors and MRSA carriage. Despite these limitations, we were able to derive and internally validate a scoring system that can be used both for the prediction of individual probability of carriage and the expected positive predictive value of a rapid diagnostic test. Due to the use of unequivocal and readily available predictors, a completely automated use of our model is conceivable. Future work will focus on the external validation of our model using data from different European hospitals.