Simulated effect of pneumococcal vaccination in the Netherlands on existing rules constructed in a non-vaccinated cohort predicting sequelae after bacterial meningitis

About 5 per 100,000 children suffer from bacterial meningitis (BM) in the Netherlands per year [1]. BM is still a serious disease with a mortality rate of 5%, despite antibiotic treatment and advances in care of critically ill patients [2]. Approximately 15-20 percent of BM survivors develop severe acute or long term sequelae such as epilepsy, hydrocephalus or hearing loss. More subtle adverse outcomes such as cognitive or behavioral limitations are present in 20-30 percent of the children [3–7]. Some of these more subtle limitations are recognized in an early phase after the BM, but a major group has substantial delay in the detection of these problems. Early detection of sequelae and adequate treatment or follow up is of major importance since it may prevent worsening of symptoms and development of secondary problems.

We have previously developed two prediction rules to identify children at risk of developing hearing loss and academic or behavioral limitations after BM. These prediction rules consist of clinical variables which can be obtained during admission. The following variables were identified as risk factors for developing hearing loss: duration of symptoms prior to admission, absence of petechiae, cerebral spinal fluid (CSF) glucose level, Streptococcus pneumoniae as causative pathogen and ataxia. The prediction rule for academic or behavioral limitations included: gender, birth weight, educational level of the father, S. pneumoniae as causative pathogen, CSF leukocyte count, delay between admission and the start of antibiotics, the use of dexamethasone, seizures, and the duration of fever during admission. The rationale and design of the studies in which the prediction rules were developed have been extensively described in earlier manuscripts [5, 6].

In both our prediction rules S. pneumoniae as causative pathogen is an important risk factor. Since June 2006 all Dutch newborn children receive a seven-valent conjugate vaccine (PCV-7) against this pathogen (serotypes 4,6B,9V,14,18C,19F,23F) on the age of 2,3,4 and 11 months.

Observations in other countries which introduced this vaccine earlier showed major decrease of BM and other infections caused by S. pneumoniae with the serotypes included in the vaccine. An increase in the incidence of the not included serotypes 19A, 6C, 22F and 15 was also observed [8–13]. As a result the population of children who might develop BM after vaccination is expected to be smaller and with different distribution of pneumococcal serotypes than the population on which the two prediction rules have been based. The aim of this study was to evaluate whether the existing prediction rules are also still applicable on a vaccinated population.

This study consists of secondary data analyses of previously collected data, used to develop the prediction rules for developing hearing loss and academic or behavioral limitations. Approval for use of this data was obtained from the entitled researchers. The construction of these rules has been described extensively in the original publications [5, 6]. In short, the design was as follows: all Dutch children born between January 1986 and December 1994 that recovered from BM caused by Neisseria meningitidis, S. pneumoniae, Streptococcus agalactiae, Escherichia coli or Listeria monocytogenes between January 1990 and December 1995 were eligible for the study. These children were selected from the files of the Netherlands Reference Laboratory for BM which collects strains and data from approximately 80% of all Dutch meningitis cases [1]. The diagnosis of BM was confirmed in CSF by culture of bacteria or by demonstrating bacterial antigens with latex agglutination. All patients with a complex onset of BM (defined as: meningitis secondary to immunodeficiency states, central nervous system surgery, cranial trauma or cerebrospinal fluid shunt infection, or relapsing meningitis), pre-existing cognitive or behavioral problems or severe handicaps were excluded [5, 6].

Sixteen hundred and five children treated in 110 different Dutch hospitals met the criteria for inclusion. The pediatricians of these hospitals were asked to send the parents (or guardians) a standard letter requesting their participation. The parents who returned informed consent were sent screening questionnaires about health, learning and behavior, and were asked permission to study the medical records of their child.

Finally, 628 children were included in the cohort regarding hearing loss and 674 children in the cohort used for the development of the rule predicting academic or behavioral limitations [5, 6].

Hearing loss was defined as a perceptive loss of >25 dB and was based on information from questionnaires and medical records. A total of 43 of the remaining 628 children were identified as having hearing loss. Information on the risk factors included in the prediction rule for hearing loss, was extracted from the medical records [5].

Academic or behavioral limitations were identified in two steps. The first step was based on parental perception on learning and behavior studied by two questionnaires. Based on the scores on the questionnaires the 674 children were divided into two groups: one with children with parents with complaints on learning and behavior (n = 134, 20%) and one with children with parents without these complaints (n = 540, 80%). For the second step a nested case-control study of 201 children was constructed. Therefore a random sample from both groups was drawn: 100 children from the group with and 101 from the group without suspected problems. The medical records of these 201 children were studied to identify the risk factors included in the prediction rule for academic or behavioral limitations. Exploration of these records sometimes provided new information on medical history, resulting in extra patients meeting exclusion criteria. Therefore another 19 of the 201 children were excluded, leaving a total of 182 children in the cohort (89 with suspected problems based on parental perception, 93 without). These 182 children were extensively tested on academic performance using a Academic Achievement Test (AAT) and on behavioral functioning with the Child Behavior Checklist (CBCL) [14]. Out of 182 children, 84 were identified as having academic or behavioral limitations, leaving 98 children without limitations [6].

Based on the two primary outcome measures "hearing loss" and "academic or behavioral limitations" and the risk factors found, the two prediction rules were developed using a multivariable logistic regression model. Potential risk factors were studied resulting, as mentioned in the introduction paragraph, in 5 independent risk factors in the hearing loss prediction model and in 9 factors in the prediction rule for academic or behavioral limitations [5, 6]. Figure 1 and 2 show patient flow charts of the inclusion and assessment.

We studied the effect of the introduction of a seven valent conjugate pneumococcal vaccine on two previous developed prediction rules for hearing loss and academic or behavioral limitations in the Netherlands. The presumed effect of vaccination was simulated by excluding all children infected by S. pneumoniae with the serotypes included in the vaccine, from both previous collected cohorts. After calculating two new prediction rules we found that all original risk factors remained significant. So it can be assumed that the original prediction rules will also be applicable on a vaccinated population. The discriminative ability of both prediction rules remained as high as observed in the previous studies of Koomen et al. [5, 6]. The risk factor "S. pneumoniae as causative pathogen" remained significant. This could be explained by the fact that the children infected by some of the non-vaccinated serotypes of this bacterium had a less favorable outcome than children infected by the PCV-7 serotypes or other pathogens [15], although there is not much evidence available for this phenomenon.

Some limitations of this study should be acknowledged. First item to address originates from the original prediction rules. From the cohort of 1605 children eligible, most children were excluded because the pediatricians or the parents did not want to participate. Although most of the basic patient characteristics as sex, age, and causative pathogen were very similar in both the original cohort and the included children, suggesting that these cohorts were representative for the original BM population, exclusion bias is possible. Further, the outcome measure "academic or behavioral limitations" was prospectively and extensively tested, but "hearing loss" was retrospectively assessed, leaving the possibility that some mild hearing problems were missed [5, 6]. These limitations of the original studies also apply to the updated rules in this manuscript.

Second, the sample sizes included in the analyses are relatively small for developing new prediction models. General rules of thumb recommend at least ten cases per included variable. So the recommend number of children with hearing loss or limitations was 50 and 90 respectively, whereas our cohorts included 31 and 75 children with hearing loss or limitations, respectively. The variable ataxia included in the rule for predicting hearing loss and the variable prolonged fever in the rule for limitations had a very high Odds Ratio and a very large confidence interval. This may reflect this power problem. As mentioned before, the regression coefficient of the variable "delay of more than 6 hours between admission and start of antibiotics" changed from 0.9 tot 2.4, with also relatively high Odds Ratio and a large confidence interval. This striking difference may also be a result from low power.

The next issue concerns the applicability of this new constructed rule in clinical practice. Although we believe that the methodology and statistical analyses used are adequate, we do realize that the original rules are not externally validated yet. It is known that prediction rules might act significantly different when applied to a new cohort, making external validation and, if necessary updating, essential [16–19]. External validation of the original models is performed, but to come to an optimal result for the new situation, a cohort with vaccinated children should be constructed. But, at this moment the number of vaccinated children in the Netherlands is relatively small and follow up time is short, making external validation a difficult procedure. Further, the acceptable proportion of children with problems that were not predicted is disputable, making it difficult to recommend cut off points of the scores in clinical practice. Regarding hearing loss, we believe it is justifiable to say that hearing loss may not be missed at all, proposing a cut off score of ≥10. Regarding academic and behavioral limitations, the proportion of positively predicted children is 87% using a cut off point of 64,75, decreasing to 63% with a higher cut off point (figure 4). We believe this is a reasonable result, but practitioners may find this score to liberal.

Final point of discussion is the fact that the vaccine does not provide full coverage, although rates differ between countries and age-groups [8, 9, 20, 21], and that serotype replacement is likely. This issue and an increase of invasive infections with non-vaccine serotypes has been observed in other countries [9–12, 22]. The actual effect of vaccination and the amount of serotype replacement are unpredictable and it can take many years before a new balance in serotype prevalence is established [12, 23]. Because BM has most impact in young children, especially those under the age of 1 year, vaccination is most relevant for this age group. It is therefore especially important to validate and update the prediction rules in this age group in future studies [16, 19]. When validated, effort should be shifted to determine how to implement the clinical use of the prediction rules and thereby improve clinically outcome for BM surviving children.

Although incidence is decreasing BM is still a serious infectious disease with high mortality and morbidity. The previously developed prediction rules showed that S. pneumoniae as causative pathogen is an important risk factor for sequelae. This study concludes that it can be assumed that the original prediction rules will also be applicable on a PCV-7 vaccinated population. However, the vaccine does not provide 100% coverage and there is evidence that serotype replacement will occur. The prediction rules need to be validated externally and further research is needed to investigate the impact of vaccination on serotype replacement.