We studied the effect of the introduction of a seven valent conjugate pneumococcal vaccine on two previous developed prediction rules for hearing loss and academic or behavioral limitations in the Netherlands. The presumed effect of vaccination was simulated by excluding all children infected by S. pneumoniae with the serotypes included in the vaccine, from both previous collected cohorts. After calculating two new prediction rules we found that all original risk factors remained significant. So it can be assumed that the original prediction rules will also be applicable on a vaccinated population. The discriminative ability of both prediction rules remained as high as observed in the previous studies of Koomen et al. [5, 6]. The risk factor " S. pneumoniae as causative pathogen" remained significant. This could be explained by the fact that the children infected by some of the non-vaccinated serotypes of this bacterium had a less favorable outcome than children infected by the PCV-7 serotypes or other pathogens , although there is not much evidence available for this phenomenon.
Some limitations of this study should be acknowledged. First item to address originates from the original prediction rules. From the cohort of 1605 children eligible, most children were excluded because the pediatricians or the parents did not want to participate. Although most of the basic patient characteristics as sex, age, and causative pathogen were very similar in both the original cohort and the included children, suggesting that these cohorts were representative for the original BM population, exclusion bias is possible. Further, the outcome measure "academic or behavioral limitations" was prospectively and extensively tested, but "hearing loss" was retrospectively assessed, leaving the possibility that some mild hearing problems were missed [5, 6]. These limitations of the original studies also apply to the updated rules in this manuscript.
Second, the sample sizes included in the analyses are relatively small for developing new prediction models. General rules of thumb recommend at least ten cases per included variable. So the recommend number of children with hearing loss or limitations was 50 and 90 respectively, whereas our cohorts included 31 and 75 children with hearing loss or limitations, respectively. The variable ataxia included in the rule for predicting hearing loss and the variable prolonged fever in the rule for limitations had a very high Odds Ratio and a very large confidence interval. This may reflect this power problem. As mentioned before, the regression coefficient of the variable "delay of more than 6 hours between admission and start of antibiotics" changed from 0.9 tot 2.4, with also relatively high Odds Ratio and a large confidence interval. This striking difference may also be a result from low power.
The next issue concerns the applicability of this new constructed rule in clinical practice. Although we believe that the methodology and statistical analyses used are adequate, we do realize that the original rules are not externally validated yet. It is known that prediction rules might act significantly different when applied to a new cohort, making external validation and, if necessary updating, essential [16–19]. External validation of the original models is performed, but to come to an optimal result for the new situation, a cohort with vaccinated children should be constructed. But, at this moment the number of vaccinated children in the Netherlands is relatively small and follow up time is short, making external validation a difficult procedure. Further, the acceptable proportion of children with problems that were not predicted is disputable, making it difficult to recommend cut off points of the scores in clinical practice. Regarding hearing loss, we believe it is justifiable to say that hearing loss may not be missed at all, proposing a cut off score of ≥10. Regarding academic and behavioral limitations, the proportion of positively predicted children is 87% using a cut off point of 64,75, decreasing to 63% with a higher cut off point (figure 4). We believe this is a reasonable result, but practitioners may find this score to liberal.
Final point of discussion is the fact that the vaccine does not provide full coverage, although rates differ between countries and age-groups [8, 9, 20, 21], and that serotype replacement is likely. This issue and an increase of invasive infections with non-vaccine serotypes has been observed in other countries [9–12, 22]. The actual effect of vaccination and the amount of serotype replacement are unpredictable and it can take many years before a new balance in serotype prevalence is established [12, 23]. Because BM has most impact in young children, especially those under the age of 1 year, vaccination is most relevant for this age group. It is therefore especially important to validate and update the prediction rules in this age group in future studies [16, 19]. When validated, effort should be shifted to determine how to implement the clinical use of the prediction rules and thereby improve clinically outcome for BM surviving children.