Simulated effect of pneumococcal vaccination in the Netherlands on existing rules constructed in a non-vaccinated cohort predicting sequelae after bacterial meningitis
© Commandeur et al; licensee BioMed Central Ltd. 2010
Received: 5 March 2010
Accepted: 3 September 2010
Published: 3 September 2010
Previously two prediction rules identifying children at risk of hearing loss and academic or behavioral limitations after bacterial meningitis were developed. Streptococcus pneumoniae as causative pathogen was an important risk factor in both. Since 2006 Dutch children receive seven-valent conjugate vaccination against S. pneumoniae. The presumed effect of vaccination was simulated by excluding all children infected by S. pneumoniae with the serotypes included in the vaccine, from both previous collected cohorts (between 1990-1995).
Children infected by one of the vaccine serotypes were excluded from both original cohorts (hearing loss: 70 of 628 children; academic or behavioral limitations: 26 of 182 children). All identified risk factors were included in multivariate logistic regression models. The discriminative ability of both new models was calculated.
The same risk factors as in the original models were significant. The discriminative ability of the original hearing loss model was 0.84 and of the new model 0.87. In the academic or behavioral limitations model it was 0.83 and 0.84 respectively.
It can be assumed that the prediction rules will also be applicable on a vaccinated population. However, vaccination does not provide 100% coverage and evidence is available that serotype replacement will occur. The impact of vaccination on serotype replacement needs to be investigated, and the prediction rules must be validated externally.
About 5 per 100,000 children suffer from bacterial meningitis (BM) in the Netherlands per year . BM is still a serious disease with a mortality rate of 5%, despite antibiotic treatment and advances in care of critically ill patients . Approximately 15-20 percent of BM survivors develop severe acute or long term sequelae such as epilepsy, hydrocephalus or hearing loss. More subtle adverse outcomes such as cognitive or behavioral limitations are present in 20-30 percent of the children [3–7]. Some of these more subtle limitations are recognized in an early phase after the BM, but a major group has substantial delay in the detection of these problems. Early detection of sequelae and adequate treatment or follow up is of major importance since it may prevent worsening of symptoms and development of secondary problems.
We have previously developed two prediction rules to identify children at risk of developing hearing loss and academic or behavioral limitations after BM. These prediction rules consist of clinical variables which can be obtained during admission. The following variables were identified as risk factors for developing hearing loss: duration of symptoms prior to admission, absence of petechiae, cerebral spinal fluid (CSF) glucose level, Streptococcus pneumoniae as causative pathogen and ataxia. The prediction rule for academic or behavioral limitations included: gender, birth weight, educational level of the father, S. pneumoniae as causative pathogen, CSF leukocyte count, delay between admission and the start of antibiotics, the use of dexamethasone, seizures, and the duration of fever during admission. The rationale and design of the studies in which the prediction rules were developed have been extensively described in earlier manuscripts [5, 6].
In both our prediction rules S. pneumoniae as causative pathogen is an important risk factor. Since June 2006 all Dutch newborn children receive a seven-valent conjugate vaccine (PCV-7) against this pathogen (serotypes 4,6B,9V,14,18C,19F,23F) on the age of 2,3,4 and 11 months.
Observations in other countries which introduced this vaccine earlier showed major decrease of BM and other infections caused by S. pneumoniae with the serotypes included in the vaccine. An increase in the incidence of the not included serotypes 19A, 6C, 22F and 15 was also observed [8–13]. As a result the population of children who might develop BM after vaccination is expected to be smaller and with different distribution of pneumococcal serotypes than the population on which the two prediction rules have been based. The aim of this study was to evaluate whether the existing prediction rules are also still applicable on a vaccinated population.
This study consists of secondary data analyses of previously collected data, used to develop the prediction rules for developing hearing loss and academic or behavioral limitations. Approval for use of this data was obtained from the entitled researchers. The construction of these rules has been described extensively in the original publications [5, 6]. In short, the design was as follows: all Dutch children born between January 1986 and December 1994 that recovered from BM caused by Neisseria meningitidis, S. pneumoniae, Streptococcus agalactiae, Escherichia coli or Listeria monocytogenes between January 1990 and December 1995 were eligible for the study. These children were selected from the files of the Netherlands Reference Laboratory for BM which collects strains and data from approximately 80% of all Dutch meningitis cases . The diagnosis of BM was confirmed in CSF by culture of bacteria or by demonstrating bacterial antigens with latex agglutination. All patients with a complex onset of BM (defined as: meningitis secondary to immunodeficiency states, central nervous system surgery, cranial trauma or cerebrospinal fluid shunt infection, or relapsing meningitis), pre-existing cognitive or behavioral problems or severe handicaps were excluded [5, 6].
Sixteen hundred and five children treated in 110 different Dutch hospitals met the criteria for inclusion. The pediatricians of these hospitals were asked to send the parents (or guardians) a standard letter requesting their participation. The parents who returned informed consent were sent screening questionnaires about health, learning and behavior, and were asked permission to study the medical records of their child.
Hearing loss was defined as a perceptive loss of >25 dB and was based on information from questionnaires and medical records. A total of 43 of the remaining 628 children were identified as having hearing loss. Information on the risk factors included in the prediction rule for hearing loss, was extracted from the medical records .
Academic or behavioral limitations were identified in two steps. The first step was based on parental perception on learning and behavior studied by two questionnaires. Based on the scores on the questionnaires the 674 children were divided into two groups: one with children with parents with complaints on learning and behavior (n = 134, 20%) and one with children with parents without these complaints (n = 540, 80%). For the second step a nested case-control study of 201 children was constructed. Therefore a random sample from both groups was drawn: 100 children from the group with and 101 from the group without suspected problems. The medical records of these 201 children were studied to identify the risk factors included in the prediction rule for academic or behavioral limitations. Exploration of these records sometimes provided new information on medical history, resulting in extra patients meeting exclusion criteria. Therefore another 19 of the 201 children were excluded, leaving a total of 182 children in the cohort (89 with suspected problems based on parental perception, 93 without). These 182 children were extensively tested on academic performance using a Academic Achievement Test (AAT) and on behavioral functioning with the Child Behavior Checklist (CBCL) . Out of 182 children, 84 were identified as having academic or behavioral limitations, leaving 98 children without limitations .
We simulated the presumed effect of the vaccination by excluding all children who were infected by S. pneumoniae with one of the seven vaccine serotypes (4,6B,9V,14,18C,19F,23F) from both cohorts. Twenty six (9 with and 17 without limitations) of 182 children of the academic or behavioral limitations cohort and 70 (12 with and 58 without hearing loss) of 628 children from the hearing loss cohort were excluded. Two new prediction rules were calculated using this two adjusted cohorts.
Univariate logistic regression analysis was used to assess the effects of all original potential risk factors collected in the adjusted cohorts. Continuous variables were dichotomized when a linear relationship with the outcome could not be assumed. All variables with a p-value of less than 0.20 were included in a multivariate logistic regression model. This first selection step was needed because the number of potential risk factors compared to the group size was too large. The prediction models were developed using a forward selection method using a threshold p-value of 0.10 for variables to be included in the models. The area under the curve (AUC) of the receiver operating curve (ROC) was calculated to assess the discriminative ability of both prediction rules. A score for each risk factor was determined by dividing the (original, not rounded) regression coefficient of each variable in the model by the smallest regression coefficient of the model, multiplied by 10 and rounding off to the nearest integer. From each child a score was calculated by filling in the values from his or her risk factors in the prediction rules. These values are summarized in a table and a cut off score for having a high risk of developing hearing loss or academic or behavioral limitations was chosen to obtain the highest discriminative ability between groups. All analyses were carried out using SPSS statistical software edition 14.0 (SPSS inc, Chicago, US).
Independent predictors of hearing loss after bacterial meningitis.
Odds ratio (95% CI)
Contribution to risk score
Duration of symptoms
< 1 day
> 2 days
Absence of petechiae
CSF glucose level < 0.6 mmol/L
Number of children with and without hearing loss across categories of the risk score. Values represent numbers (percentages).
Children in the cohort n = 558(%), cumulative n(%)
Children with hearing loss n = 31 (%), cumulative n(%)
Children without hearing loss n = 527 (%), cumulative n(%)
Academic or behavioral limitations
Independent predictors of academic or behavioral limitations after bacterial meningitis.
Odds ratio (95% CI)
Contribution to risk score
Birth weight ≤3000 gram
Educational level fathera
S. pneumoniae as causative pathogen
CSF leukocyte count/10000 cells (/3 mm3)
Delay of more than 6 hours between admission and start of antibiotics
≤ 2 days
> 2 days
Seizures treated with anticonvulsive therapy
Prolonged fever (> 9 days)
Number of children in the cohort with and without academic or behavioral limitations across categories of the risk score. Values represent numbers (percentages).
Children in the cohort n = 166(%), cumulative n(%)
Children with academic or behavioral limitations n = 75(%), cumulative n(%)
Children without academic or behavioral limitations n = 91(%), cumulative n(%)
> 29 & ≤64,75
> 64,75 &≤94,66
> 94,66 & ≤134,5
We studied the effect of the introduction of a seven valent conjugate pneumococcal vaccine on two previous developed prediction rules for hearing loss and academic or behavioral limitations in the Netherlands. The presumed effect of vaccination was simulated by excluding all children infected by S. pneumoniae with the serotypes included in the vaccine, from both previous collected cohorts. After calculating two new prediction rules we found that all original risk factors remained significant. So it can be assumed that the original prediction rules will also be applicable on a vaccinated population. The discriminative ability of both prediction rules remained as high as observed in the previous studies of Koomen et al. [5, 6]. The risk factor "S. pneumoniae as causative pathogen" remained significant. This could be explained by the fact that the children infected by some of the non-vaccinated serotypes of this bacterium had a less favorable outcome than children infected by the PCV-7 serotypes or other pathogens , although there is not much evidence available for this phenomenon.
Some limitations of this study should be acknowledged. First item to address originates from the original prediction rules. From the cohort of 1605 children eligible, most children were excluded because the pediatricians or the parents did not want to participate. Although most of the basic patient characteristics as sex, age, and causative pathogen were very similar in both the original cohort and the included children, suggesting that these cohorts were representative for the original BM population, exclusion bias is possible. Further, the outcome measure "academic or behavioral limitations" was prospectively and extensively tested, but "hearing loss" was retrospectively assessed, leaving the possibility that some mild hearing problems were missed [5, 6]. These limitations of the original studies also apply to the updated rules in this manuscript.
Second, the sample sizes included in the analyses are relatively small for developing new prediction models. General rules of thumb recommend at least ten cases per included variable. So the recommend number of children with hearing loss or limitations was 50 and 90 respectively, whereas our cohorts included 31 and 75 children with hearing loss or limitations, respectively. The variable ataxia included in the rule for predicting hearing loss and the variable prolonged fever in the rule for limitations had a very high Odds Ratio and a very large confidence interval. This may reflect this power problem. As mentioned before, the regression coefficient of the variable "delay of more than 6 hours between admission and start of antibiotics" changed from 0.9 tot 2.4, with also relatively high Odds Ratio and a large confidence interval. This striking difference may also be a result from low power.
The next issue concerns the applicability of this new constructed rule in clinical practice. Although we believe that the methodology and statistical analyses used are adequate, we do realize that the original rules are not externally validated yet. It is known that prediction rules might act significantly different when applied to a new cohort, making external validation and, if necessary updating, essential [16–19]. External validation of the original models is performed, but to come to an optimal result for the new situation, a cohort with vaccinated children should be constructed. But, at this moment the number of vaccinated children in the Netherlands is relatively small and follow up time is short, making external validation a difficult procedure. Further, the acceptable proportion of children with problems that were not predicted is disputable, making it difficult to recommend cut off points of the scores in clinical practice. Regarding hearing loss, we believe it is justifiable to say that hearing loss may not be missed at all, proposing a cut off score of ≥10. Regarding academic and behavioral limitations, the proportion of positively predicted children is 87% using a cut off point of 64,75, decreasing to 63% with a higher cut off point (figure 4). We believe this is a reasonable result, but practitioners may find this score to liberal.
Final point of discussion is the fact that the vaccine does not provide full coverage, although rates differ between countries and age-groups [8, 9, 20, 21], and that serotype replacement is likely. This issue and an increase of invasive infections with non-vaccine serotypes has been observed in other countries [9–12, 22]. The actual effect of vaccination and the amount of serotype replacement are unpredictable and it can take many years before a new balance in serotype prevalence is established [12, 23]. Because BM has most impact in young children, especially those under the age of 1 year, vaccination is most relevant for this age group. It is therefore especially important to validate and update the prediction rules in this age group in future studies [16, 19]. When validated, effort should be shifted to determine how to implement the clinical use of the prediction rules and thereby improve clinically outcome for BM surviving children.
Although incidence is decreasing BM is still a serious infectious disease with high mortality and morbidity. The previously developed prediction rules showed that S. pneumoniae as causative pathogen is an important risk factor for sequelae. This study concludes that it can be assumed that the original prediction rules will also be applicable on a PCV-7 vaccinated population. However, the vaccine does not provide 100% coverage and there is evidence that serotype replacement will occur. The prediction rules need to be validated externally and further research is needed to investigate the impact of vaccination on serotype replacement.
- Netherlands Reference Laboratory for Bacterial Meningitis: Bacterial meningitis in the Netherlands: annual report 2006. 2007, Amsterdam, The Netherlands: AMC/RIVM
- Saez-Llorens X, McCracken GH: Bacterial meningitis in children. Lancet. 2003, 361: 2139-2148. 10.1016/S0140-6736(03)13693-8.View ArticlePubMed
- Baraff LJ, Lee SI, Schriger DL: Outcomes of bacterial meningitis in children: a meta-analysis. Pediatr Infect Dis J. 1993, 12: 389-394. 10.1097/00006454-199305000-00008.View ArticlePubMed
- Grimwood K, Anderson VA, Bond L, Catroppa C, Hore RL, Keir EH, Nolan T, Roberton DM: Adverse outcomes of bacterial meningitis in school-age survivors. Pediatrics. 1995, 95: 646-656.PubMed
- Koomen I, Grobbee DE, Roord JJ, Donders R, Jennekens-Schinkel A, van Furth AM: Hearing loss at school age in survivors of bacterial meningitis: assessment, incidence, and prediction. Pediatrics. 2003, 112: 1049-1053. 10.1542/peds.112.5.1049.View ArticlePubMed
- Koomen I, Grobbee DE, Roord JJ, Jennekens-Schinkel A, van der Lei HD, Kraak MA, van Furth AM: Prediction of academic and behavioural limitations in school-age survivors of bacterial meningitis. Acta Paediatr. 2004, 93: 1378-1385. 10.1111/j.1651-2227.2004.tb02939.x.View ArticlePubMed
- Lutsar I, Siirde T, Soopold T: Long term follow-up of Estonian children after bacterial meningitis. Pediatr Infect Dis J. 1995, 14: 624-625. 10.1097/00006454-199507000-00014.View ArticlePubMed
- Center KJ: Prevenar vaccination: review of the global data, 2006. Vaccine. 2007, 25: 3085-3089. 10.1016/j.vaccine.2007.01.021.View ArticlePubMed
- Dias R, Canica M: Invasive pneumococcal disease in Portugal prior to and after the introduction of pneumococcal heptavalent conjugate vaccine. FEMS Immunol Med Microbiol. 2007, 51: 35-42. 10.1111/j.1574-695X.2007.00283.x.View ArticlePubMed
- Jacobs MR, Good CE, Bajaksouzian S, Windau AR: Emergence of Streptococcus pneumoniae serotypes 19A, 6C, and 22F and serogroup 15 in Cleveland, Ohio, in relation to introduction of the protein-conjugated pneumococcal vaccine. Clin Infect Dis. 2008, 47: 1388-1395. 10.1086/592972.View ArticlePubMed
- Mera R, Miller LA, Fritsche TR, Jones RN: Serotype replacement and multiple resistance in Streptococcus pneumoniae after the introduction of the conjugate pneumococcal vaccine. Microb Drug Resist. 2008, 14: 101-107. 10.1089/mdr.2008.0782.View ArticlePubMed
- Pletz MW, Maus U, Krug N, Welte T, Lode H: Pneumococcal vaccines: mechanism of action, impact on epidemiology and adaption of the species. Int J Antimicrob Agents. 2008
- Poehling KA, Talbot TR, Griffin MR, Craig AS, Whitney CG, Zell E, Lexau CA, Thomas AR, Harrison LH, Reingold AL, et al: Invasive pneumococcal disease among infants before and after introduction of pneumococcal conjugate vaccine. Jama. 2006, 295: 1668-1674. 10.1001/jama.295.14.1668.View ArticlePubMed
- Achenbach TM, McConaughy SH: Empirically Based assessment of Child and Adolescent Psychopathology. 1997, Thousand Oaks, CA: Sage
- Jansen AG, Rodenburg GD, van der Ende A, van Alphen L, Veenhoven RH, Spanjaard L, Sanders EA, Hak E: Invasive pneumococcal disease among adults: associations among serotypes, disease characteristics, and outcome. Clin Infect Dis. 2009, 49: e23-29. 10.1086/600045.View ArticlePubMed
- Altman DG, Royston P: What do we mean by validating a prognostic model?. Stat Med. 2000, 19: 453-473. 10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5.View ArticlePubMed
- Altman DG, Vergouwe Y, Royston P, Moons KG: Prognosis and prognostic research: validating a prognostic model. Bmj. 2009, 338: b605-10.1136/bmj.b605.View ArticlePubMed
- Bleeker SE, Moll HA, Steyerberg EW, Donders AR, Derksen-Lubsen G, Grobbee DE, Moons KG: External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003, 56: 826-832. 10.1016/S0895-4356(03)00207-5.View ArticlePubMed
- Toll DB, Janssen KJ, Vergouwe Y, Moons KG: Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008, 61: 1085-1094. 10.1016/j.jclinepi.2008.04.008.View ArticlePubMed
- Black S, Shinefield H, Fireman B, Lewis E, Ray P, Hansen JR, Elvin L, Ensor KM, Hackell J, Siber G, et al: Efficacy, safety and immunogenicity of heptavalent pneumococcal conjugate vaccine in children. Northern California Kaiser Permanente Vaccine Study Center Group. Pediatr Infect Dis J. 2000, 19: 187-195. 10.1097/00006454-200003000-00003.View ArticlePubMed
- Vestrheim DF, Lovoll O, Aaberge IS, Caugant DA, Hoiby EA, Bakke H, Bergsaker MR: Effectiveness of a 2+1 dose schedule pneumococcal conjugate vaccination programme on invasive pneumococcal disease among children in Norway. Vaccine. 2008, 26: 3277-3281. 10.1016/j.vaccine.2008.03.087.View ArticlePubMed
- Invasive pneumococcal disease in children 5 years after conjugate vaccine introduction--eight states, 1998-2005. MMWR Morb Mortal Wkly Rep. 2008, 57: 144-148.
- Hammitt LL, Bruden DL, Butler JC, Baggett HC, Hurlburt DA, Reasonover A, Hennessy TW: Indirect effect of conjugate vaccine on adult carriage of Streptococcus pneumoniae: an explanation of trends in invasive pneumococcal disease. J Infect Dis. 2006, 193: 1487-1494. 10.1086/503805.View ArticlePubMed
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2334/10/259/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.