This article has Open Peer Review reports available.
Selection of key recommendations for quality indicators describing good quality outbreak response
© Belfroid et al.; licensee BioMed Central. 2015
Received: 29 September 2014
Accepted: 12 March 2015
Published: 31 March 2015
The performance of recommended control measures is necessary for quick and uniform infectious disease outbreak control. To assess whether these procedures are performed, a valid set of quality indicators (QIs) is required. The goal of this study was to select a set of key recommendations that can be systematically translated into QIs to measure the quality of infectious disease outbreak response from the perspective of disaster emergency responders and infectious disease control professionals.
Applying the Rand modified Delphi procedure, the following steps were taken to systematically select a set of key recommendations: extraction of recommendations from relevant literature; appraisal of the recommendations in terms of relevance through questionnaires to experts; expert meeting to discuss recommendations; prioritization of recommendations through a second questionnaire; and final expert meeting to approve the selected set. Infectious disease physicians and nurses, policymakers and communication experts participated in the expert group (n = 48).
In total, 54 national and international publications were systematically searched for recommendations, yielding over 200 recommendations. The Rand modified Delphi procedure resulted in a set of 65 key recommendations. The key recommendations were categorized into 10 domains describing the whole response pathway from outbreak recognition to aftercare.
This study provides a set of key recommendations that represents ‘good quality of response to an infectious disease outbreak’. These key recommendations can be systematically translated into QIs. Organizations and professionals involved in outbreak control can use these QIs to monitor the quality of response to infectious disease outbreaks and to assess in which domains improvement is needed.
Infectious disease outbreaks are a global threat to public health that can have a high economical and societal impact [1,2]. During an outbreak, recommended control measures need to be timely and uniformly performed to curb the spread of pathogens and, ultimately, to reduce the number of persons becoming infected .
Numerous documents -from various countries or regions addressing various types of outbreaks- describe such recommended or ‘good quality’ response to infectious disease outbreaks. Unfortunately, publication and dissemination of scripts, guidelines and scientific advice does not guarantee good quality outbreak response [4-6]. Evaluations of recent crises show large variation in the implementation of crisis advice and thus in the quality of outbreak response. A Dutch study examining the response to the Q-fever outbreak showed large regional variance in the implementation of the nationally advised measures . Similarly, evaluation of the control measures during the early stages of the lymphogranuloma venereum (LGV) outbreak in the various EU countries showed great differences between countries with respect to case definitions, laboratory testing and antimicrobial drug treatment . A study examining the causes of differences in the implementation of crisis advice showed that among others, the various categories of professionals involved in outbreak control lacked clearly defined measures to monitor the execution of ‘key actions’ .
Quality indicators (QIs) can be used to gain insight into the quality of response and, even more importantly, they can be used to measure the effects of interventions aimed at improving response . In this manner, QIs provide a tool to systematically monitor response quality. Two studies developed QIs for infectious disease outbreak response describing the major domains of outbreak response [11,12]. Both studies lack transparency and reproducibility because they do not provide insight into among others; the selected literature, selected experts and the definition of consensus. It is important that QIs are developed in a systematic and transparent way [10,13,14], which to our knowledge has not been done for infectious disease outbreaks.
The goal of this study was therefore to select a set of key recommendations that can be systematically translated into QIs to measure the quality of infectious disease outbreak response from the perspective of disaster emergency responders and infectious disease control professionals (ID-control professionals) using a valid development procedure. We aimed to develop a generic set of key recommendations that can be used to measure the quality of outbreak response, irrespective of the causative pathogen.
We used the systematic RAND modified Delphi method to develop and select -in a multistep approach (see Step1 through 5 below)- a set of key recommendations representing good quality infectious disease outbreak response . In this iterative method the individual opinion of experts is aggregated into group consensus. Recommendations for infectious disease outbreak response were extracted from the literature, and presented to a multidisciplinary expert panel. The panel achieved consensus on a set of key recommendations during two questionnaire rounds and two face-to face consensus meetings. Formal ethical approval from a medical ethical committee was not required for this research in the Netherlands since it does not entail subjecting participants to medical treatment or imposing specific rules of conduct on participants. All the experts consented to participate in the study and were aware that their responses would be used for research purposes.
Step 1 – Literature search and extraction of recommendations
Medline search strategy
The gold standard search strategy of the Cochrane Effective Practice and Organisation of Care group (http://epoc.cochrane.org/)
Communicable diseases OR
disease outbreaks OR
Quality Assessment* OR
Quality Measure* OR
Risk Assessment OR
Contact Tracing OR
Disease notification OR
Disease Management OR
Infection Control OR
Mandatory Testing OR
Universal Precautions OR
Scientific AND Advice
Population Surveillance OR
Severe Acute Respiratory
Next, we collected grey literature, i.e. Dutch documents on good quality response including national guidance, national outbreak advice, contracts between health care organizations and disaster care plans. We also included national evaluations of recent infectious disease crises such as Q-fever and the 2009 flu pandemic. The inclusion of grey literature was made on the basis of recommendations from national specialists on infectious disease preparedness- and control who were asked to judge appropriateness.
Two researchers (EB and MHi) performed the extraction of recommendations independently on a sample consisting of 25% of all selected sources (literature review and grey literature). The researchers extracted good quality response recommendations from the selected literature. Discrepancies between the two researchers were discussed until consensus was reached. After reaching consensus on this 25% sample, one researcher continued to extract recommendations from the remaining selected literature (EB). The two researchers examined the total set of recommendations to remove identical recommendations.
All recommendations were discussed with the main researchers involved in this study (MHu, JH, AT, EB) in two meetings. In these meetings we selected in consensus and while applying the inclusion criteria, existing generic recommendations or aggregated pathogen and disease specific recommendations, which were subsequently presented to the participants in the expert panels during the next stages.
Step 2 – First questionnaire round
The consensus procedure took place between September 2012 and May 2013. We approached all 25 public health regions from the Netherlands by e-mail and invited public health infectious disease experts from their region to participate in the expert group. Our expert panels consisted of 48 Dutch experts in public health (28 ID-control professionals and 20 disaster emergency responders) who all had experience in the preparedness and/or control of an infectious disease outbreak. All regions were represented.
Two digital Limesurvey (a digital open source survey application) questionnaires were composed, one for the ID-control professionals and one for the disaster emergency responders. In this process, recommendations were assigned to the responsible organization in the Netherlands: logistical support recommendations were presented to disaster emergency responders and infectious disease control recommendations were presented to ID-control professionals. In the Netherlands, the disaster emergency responder tasks lay with charge of logistical support of outbreak control while the coordination of outbreak measures is a responsibility of the ID-control experts. This, of course, can be different in other countries.
Both expert groups followed a parallel, methodologically identical path: each expert group assessed the recommendations regarding their expertise on relevance. Relevance was graded by the experts in response to the following question; “To what extent do you consider this recommendation as a relevant element for measuring the quality of infectious disease outbreak response?” on a 9-point Likert scale (1 = totally disagree, 9 = totally agree). Experts could comment on recommendations and could add recommendations. Recommendations were accepted or further processed based on the RAND/University of California at Los Angeles agreement criteria . Relevance scores were calculated for each item. If the recommendation had a median of 8 or 9 and >70% of the experts scored in the top tertile, then the recommendation was marked as “accepted”. If the recommendation had a median <8 and <70% scored in the top tertile, then the recommendation was marked as “not accepted” and was excluded. If the recommendation had a median <8 and >70% of the experts scored in the top tertile or the median was 8 or 9 and <70% of the experts scored in the top tertile, then the recommendation was marked as “to be discussed”.
In the second part of the questionnaire we checked whether we assigned recommendations to the correct responsible organization (disaster emergency responders or ID-control experts). If more than 33% of the responders questioned the attribution of responsibility for the action to a certain party, then the recommendation was presented to the responsible organization in the second questionnaire round.
Step 3 – Consensus meeting
Both expert groups were invited for a separate face-to-face consensus meeting in October 2012. The results of the analysis of the questionnaires were sent to the experts in advance of the consensus meeting. During the meeting, the expert panels could comment on recommendations with the label “to be discussed”. As a result, the discussed recommendations were found to be not relevant or were modified and found relevant.
Step 4 – Second questionnaire round
In step 4, two responsible organization specific questionnaires were composed of all recommendations. To categorize the recommendations, we combined two frameworks which incorporate the main domains of emergency response [16,17]. The recommendations were categorized into 35 categories (25 for ID-control experts and 10 for disaster emergency responders) which described each step in the process of infectious disease outbreak response. These 35 categories or subdomains represented 10 main domains: “Scale of the outbreak and epidemiology”, “Control measures”, “Diagnostics”, “Logistic support”, “Aftercare and conclusion”, “Communication”, “Logistics”, “Upscaling”, “Coordination of the chain of outbreak control” and “Continuity of care”.
Corrections on the assigned responsible organization based on the first questionnaire were processed in this second questionnaire. Doubles were removed if there was too much resemblance of the recommendations within one category. If a number of recommendations represented the same recommendation, but for different patient groups (for example, confirmed case, contact of cases) the recommendations were merged. We asked both expert groups to prioritize the most important recommendation per category, and we calculated the percentage of experts that selected a recommendation. If more than 15 percent of the experts selected a recommendation, the recommendation was considered prioritized. Each group only prioritized recommendations regarding their own expertise.
Step 5 – Second consensus meeting
In a final, combined face-to-face expert meeting the experts were presented the combined selected sets of quality indicators from step 4. Experts were asked to judge the completeness of the selected recommendations. As a result, recommendations were textually modified, merged or deleted.
Step 1 – Literature search and selection of recommendations
Step 2 – First questionnaire round
Forty-four experts returned the first questionnaire (response rate 92%) containing the 226 recommendations resulting from step 1. After applying the RAND/University of California at Los Angeles agreement criteria, the first consensus round resulted in 138 “accepted” recommendations. Thirty-eight recommendations were found “not accepted”, and therefore excluded. Thirty-one recommendations were marked as “to be further processed in the second questionnaire”.
Both the “accepted” recommendations and the “to be further processed in the second questionnaire” recommendations (N = 169) were subsequently included in the second questionnaire (discussed further in step 4). Nineteen recommendations were marked “ to be discussed ”, meaning that the experts did not reach consensus on the relevance or non-relevance of the items. These recommendations were further assessed in step 3.
Step 3 – Consensus meeting
In the third step, the expert panels could comment on recommendations with the label “to be discussed” in a consensus meeting. Fourteen ID-control professionals (14 out of 28) and 16 disaster emergency responders (16 out of 20) attended the consensus meeting. The ID-control professionals discussed nine recommendations, of which one was found not relevant, seven were found relevant after textual modification and one was found relevant without modification.
The disaster emergency responders discussed ten recommendations of which two were found not relevant. Two recommendations were combined into one and as a result, seven recommendations were found relevant after textual modification.
In total fifteen recommendations were accepted for further assessment.
Step 4 - Second questionnaire round
Recommendations were combined into one if the recommendations represented the same action but a different target group (for instance management of suspected case, case, confirmed case). This resulted in a reduction from 184 to 160 recommendations. The prioritization questionnaire consisted of 160 recommendations, of which 31 had to be appraised by both the disaster emergency responders and the ID-control professionals as in practice both could be made responsible for performing the recommended activity.
Forty experts returned the second questionnaire (response rate 88.9%). As a result, the ID-control professionals selected 55 recommendations, the disaster emergency responders 21. In some cases both types of experts prioritized the same recommendation. Deleting those resulted in 69 unique recommendations from both expert groups.
Step 5 – Second consensus meeting
Selected key recommendations*
Scale of the outbreak and epidemiology
The [RESPONSIBLE ORGANISATION] collects data about all cases and contacts of cases for inclusion in the case register of the [RESPONSIBLE ORGANISATION] if the nature and phase of the outbreak are such that the data are relevant to outbreak control
Scale of the outbreak and epidemiology
The [RESPONSIBLE ORGANISATION] reports the data about the index case, the epidemiological situation (number and nature of the contacts, possible source, etc.) to the National Authority in charge of infectious disease control for national surveillance and coordination
The [RESPONSIBLE ORGANISATION] is in charge of supplying prophylaxis for the designated groups
The [RESPONSIBLE ORGANISATION] informs the supervisory pharmacy about delivery of the prophylactic medication by the [RESPONSIBLE ORGANISATION]. The pharmacist helps set up the medication management system
The [RESPONSIBLE ORGANISATION] must register the indications and contraindications of the prophylactic medication by name and national insurance number of the person receiving treatment whenever the [RESPONSIBLE ORGANISATION] provides prophylaxis
The [RESPONSIBLE ORGANISATION] informs the supervising pharmacy about any delivery of medicaments by the [RESPONSIBLE ORGANISATION]. The pharmacist helps set up the medication management system
The [RESPONSIBLE ORGANISATION] must register the indications and contraindications for the medication for treatment by name and national insurance number of the person receiving treatment when the [RESPONSIBLE ORGANISATION] is in charge of the treatment
The [RESPONSIBLE ORGANISATION] vaccinates only the groups with an indication for vaccination in as far as the [RESPONSIBLE ORGANISATION] gives the vaccination
The [RESPONSIBLE ORGANISATION] obtains a list of the residents in the outbreak area (the target population) and approaches them whenever this is necessary**
The [RESPONSIBLE ORGANISATION] informs the supervisory pharmacy about any vaccine delivery by the [RESPONSIBLE ORGANISATION]. The pharmacist helps set up the medication management system
The [RESPONSIBLE ORGANISATION] looks after registering the indications and contraindications for the vaccine by name and national insurance number of the person receiving vaccination whenever the [RESPONSIBLE ORGANISATION] gives the vaccinations
The [RESPONSIBLE ORGANISATION] organises quarantine for contacts of cases, people belonging to the risk group, and other healthcare professionals, with adherence to the national guidelines
The [RESPONSIBLE ORGANISATION] verifies whether quarantine for contacts of cases, people belonging to the risk group, and other healthcare professionals was carried out with adherence to the national guidelines
Within the agreed time span, the [RESPONSIBLE ORGANISATION] must make available a fully equipped quarantine facility that meets the current requirements starting from the moment that the need of such a facility was made known
With adherence to the national guidelines, the [RESPONSIBLE ORGANISATION] places cases and people with symptoms in isolation
The [RESPONSIBLE ORGANISATION] verifies whether the national guidelines were adhered to when cases and people with symptoms were placed in isolation at home
The [RESPONSIBLE ORGANISATION] is in charge of monitoring the medical condition of any suspected case and contacts of cases by telephone and asks the person about his/her symptoms with a view to the causative pathogen. Thus quick action can be taken, should there be any indication that this person has been infected
The [RESPONSIBLE ORGANISATION] instructs people exposed to infection to register themselves with the appropriate caretaker if they develop any symptoms (e.g. flu-like symptoms such as fever and conjunctivitis)
The [RESPONSIBLE ORGANISATION] ensures that people with symptoms, people belonging to the risk group, contacts of cases, and other healthcare professionals receive instruction about the appropriate measures to prevent infection in consultation with the treating medical practitioner
The [RESPONSIBLE ORGANISATION] instructs the general population about the appropriate measures for controlling infection
The [RESPONSIBLE ORGANISATION] instructs the carrier how to transport cases in conformance with the national guidelines
The [RESPONSIBLE ORGANISATION] instructs its employees to take the prescribed protection measures the way the national guidelines recommend
The [RESPONSIBLE ORGANISATION] informs other health care professionals about personal protection measures conforming to the national guidelines
The [RESPONSIBLE ORGANISATION] checks whether its employees use the prescribed protection measures the way the national guidelines recommend
The [RESPONSIBLE ORGANISATION] provides for the correct diagnostics for cases, contacts of cases, people belonging to the risk groups, and people who meet the case definition criteria as determined in the national guidelines
The [RESPONSIBLE ORGANISATION] makes arrangements with general practitioners (GPs) and the local medical microbiology laboratory for routing the diagnostics
The [RESPONSIBLE ORGANISATION] takes charge of diagnostics and any additional control measures (conferring with the consultant microbiologist and the GP)
The [RESPONSIBLE ORGANISATION] confers with the consultant microbiologist of the region in the event of an infectious disease outbreak
The [RESPONSIBLE ORGANISATION] provides storage of medical materials (e.g. medicines, medical equipment, necessary research materials, and sterile materials) if national instructions to do so are given
The [RESPONSIBLE ORGANISATION] distributes medical materials if national instructions to do so are given
The [RESPONSIBLE ORGANISATION] provides triage based on the national criteria whenever there is a medical indication for such triage
The [RESPONSIBLE ORGANISATION] provides instruction to the medical interagency partners about the triage criteria
The [RESPONSIBLE ORGANISATION] organises specific areas (high-risk zones) for the transport, assessment, and management of suspected or confirmed cases with adherence to current protocols (this monitoring measure concerns the separation of patient flows)
The [RESPONSIBLE ORGANISATION] has direct access to diagnostic material
The [RESPONSIBLE ORGANISATION] documents where diagnostic materials can be ordered
The [RESPONSIBLE ORGANISATION] continuously has a supply of the necessary materials in stock, in conformance to the criteria, and determines where these materials can be ordered
The [RESPONSIBLE ORGANISATION], the fire department, and the police organise fire safety, security, and traffic flow around mass-meeting sites, in conformance with the mass vaccination plan
Aftercare and conclusion
When the crisis is over, each [RESPONSIBLE ORGANISATION] evaluates the actions that the organisation itself initiated
Aftercare and conclusion
During the outbreak or soon afterwards, the [RESPONSIBLE ORGANISATION] take the initiative to involve all parties concerned in setting up a plan for aftercare
The [RESPONSIBLE ORGANISATION] provides health education for people with symptoms, people belonging to the risk group, cases, and contacts of cases
The [RESPONSIBLE ORGANISATION] limits the number of people visiting infected or suspect locations, such as businesses and student homes, as much as possible
The [RESPONSIBLE ORGANISATION] adheres to the national communication guidelines
The [RESPONSIBLE ORGANISATION] has a single leader and centrally directed communication during an outbreak
The [RESPONSIBLE ORGANISATION] posts up-to-date advice, FAQs, and associated hygiene advice on its website (for residents unfamiliar with the local language as well, if relevant) or states where this advice can be found
Among themselves, the [RESPONSIBLE ORGANISATIONS] clearly mark out who is responsible for coordinating and carrying out tasks in the context of infectious disease control
The [RESPONSIBLE ORGANISATION] records the decision-making process
The [RESPONSIBLE ORGANISATION] adheres to the division of tasks between the [RESPONSIBLE ORGANISATION] and the National Authority in charge of infectious disease control
The [RESPONSIBLE ORGANISATION] makes and updates a prognosis of bottlenecks on the basis of gaps in care, the attack rate, and the specific pandemic groups
The [RESPONSIBLE ORGANISATION] works with adherence to the current procedure strategy
The [RESPONSIBLE ORGANISATIONS] inform each other in a timely way about signals and events that may lead to an infectious disease crisis
The [RESPONSIBLE ORGANISATION] appropriately upscales if and when an infectious disease crisis occurs
The Director of Public Health directs and coordinates the elements of the clinical pathway
The [RESPONSIBLE ORGANISATION] makes an appraisal based on complexity (safety partners) and capacities (of the supply chain partners) to scale with adherence to the Coordinated Regional Incident Procedure
The [RESPONSIBLE ORGANISATION] composes a regional multidisciplinary team during a potential outbreak situation
The [RESPONSIBLE ORGANISATION] presents an analysis of expected bottlenecks in regional care to the interagency medical and administrative partners of the multidisciplinary consultation team
Coordination of the chain of outbreak control
The physician in charge of infectious disease control handles communications to external colleagues about field-specific aspects, in as far as the [RESPONSIBLE ORGANISATION] is in charge of communication
Coordination of the chain of outbreak control
The [RESPONSIBLE ORGANISATION] agrees with all parties involved on a contact person for people with symptoms, people belonging to the risk group, cases, contacts of cases, and other healthcare providers
Coordination of the chain of outbreak control
If and when the National Outbreak Management Team has convened, the [RESPONSIBLE ORGANISATION] organises a meeting with the regional crisis centre to discuss the results of the Administrative Coordination Consultation that are made known by the National Authority in charge of infectious disease control
Coordination of the chain of outbreak control
The [RESPONSIBLE ORGANISATIONS] maintain permanent contact with each other
Coordination of the chain of outbreak control
The [RESPONSIBLE ORGANISATION] ensures from the beginning that communication with all parties involved is clear, complete, and timely
Coordination of the chain of outbreak control
In the early stages of an outbreak, the [RESPONSIBLE ORGANISATION] organises a face-to-face consultation with all the care partners. During this consultation, the expectations of each organisation are expressed and the assignment of roles and tasks is determined
Continuity of care
During an infectious disease crisis, the [RESPONSIBLE ORGANISATION] maintains an overview of the current need for care and care capacity
Continuity of care
The [RESPONSIBLE ORGANISATION] consults with local GPs about the coordination of general practice care (24/7)
Continuity of care
In coordination with the local care provider networks (for example GP networks and ambulance networks), the [RESPONSIBLE ORGANISATION] consults with GPs about how long the upscaled organisation of primary care will still suffice and from what time supplemental packages will be necessary to guarantee continuity
Continuity of care
In coordination with the local care provider networks (for example GP networks and ambulance networks), the [RESPONSIBLE ORGANISATION] alerts all healthcare providers regarding continuation of care and advises them to act in conformance with the agreements (e.g. for home care and nursing-and-care settings)
In this study, we selected a set of 65 key recommendations describing good quality of infectious disease outbreak response, based on scientific and grey literature, while applying the systematic Rand modified Delphi procedure. We selected key recommendations for the broad range of tasks in infectious disease outbreak response, from “control measures” and “diagnostics” to processes such as “coordination of the chain of outbreak control”. These recommendations can be systematically translated into a set of quality indicators. This QI set is a valuable tool to monitor the quality of response to infectious disease outbreak irrespective of the pathogen, and to assess in which domains improvement is needed. To our knowledge, no generic QIs have been systematically developed to measure the quality of infectious disease outbreak response from the perspective of disaster emergency responders and ID-control professionals describing the whole process of infectious disease outbreak response.
Two studies previously developed QIs applicable to some specific components of outbreak response [11,12]. One study focused on organizational aspects like vaccine availability, communication and reporting , while the other study focused on the correct and timely detection of the first cases and the initiation of prophylaxis, education and advice to healthcare workers . Although there is some resemblance between the domains selected in these two studies and our study, our key recommendations are more detailed and specific, which is crucial for a valid and reliable assessment of quality but also for selecting targets for improvement.
Our study has some strengths and limitations. The strength is that our panels consisted of 26 and 18 experts with ample experience in outbreak control, ensuring optimal face validity of the key recommendations. Literature describes that 7–15 experts per panel are needed in order to develop a reliable set of indicators, . The experts brought expertise from various fields such as: infectious disease control, policy making, public health administration, contingency planning, public health nursing, and had been involved in regional and national meetings regarding infectious disease control. Diversity of expert panel members leads to consideration of different perspectives and a wider range of alternatives .
Preferably, key recommendations are selected combining evidence- and consensus based recommendations, with a strong preference for evidence based . A limitation of our study is that, from the 656 recommendations which served as a basis for the first questionnaire round, only 35 were extracted from the international literature. In addition, the recommendations are merely practice and expert opinion based. Expert opinion is considered to be the lowest degree of evidence . This is in line with Yeager et al., who studied the literature and concluded that “the public health emergency preparedness literature is dominated by nonempirical studies”. They also describe how only 15 percent of scientific literature on infectious disease outbreaks concerns the response to outbreaks . Although this explains the lack of evidence based recommendations in our study, at the same time it stresses the importance of building a knowledge base for ‘good quality’ response to infectious disease outbreaks. Our set of key could be the starting point for such methodologically sound empirical studies. Another limitation of our study is that our results are, for pragmatic reasons, based on judgments by Dutch experts and partly on Dutch grey literature which may reduce transferability to other countries. To reduce this risk as much as possible, we used a large group of experienced experts with various backgrounds. In addition, a considerable part of the Dutch grey literature is based on documents issued by supranational institutions (e.g. the WHO checklist, ECDC or CDC guidelines). We therefore assume that our key recommendations describe essential elements of outbreak response and will have an added value in improving efforts to deliver high quality response to outbreaks. These key recommendations are formulated in such a way that they allow for adaptation to fit the specific organizational structure of different countries. Most countries have their own national guidelines. European countries lacking national guidelines are very likely to use ECDC guidance instead (such as the RAGIDA guidelines or guidance disseminated through Rapid Risk Assessments).
In conclusion, this study provides systematically selected key recommendations for good quality of infectious disease outbreak response. These recommendations can be systematically translated into QI to measure the quality of infectious disease outbreak response and to assess in which domains improvement is needed. Their consequent application will provide public health organizations with knowledge on where improvement of outbreak response is needed and how to prioritise the efforts to achieve optimal and uniform outbreak control, in order to be prepared for the next crisis.
The most successful indicators for quality improvement are indicators that are measurable, have potential for feasible improvement, are capable of detecting differences in scores and therefore of discriminating between organizations and are applicable to a large part of the population . At this moment, we are performing a study to test the measurability of our set of key recommendations in practice.
The authors are grateful to all experts that participated in the consensus procedure for sharing their knowledge and practical experience on the subject. We thank the advisory group and consultative group for providing useful guidance for conducting the study, with special thanks to Prof. Koos van der Velden and Toos Waegemaekers. We thank Toos Waegemaekers and Sjaak de Gouw for chairing the consensus meetings. This work was supported by the Netherlands Organisation for Health Research and Development (ZonMw) who provided a research grant [grant number 204000003] to the authors.
- Brooke RJ, Van Lier A, Donker GA, Van Der Hoek W, Kretzschmar ME. Comparing the impact of two concurrent infectious disease outbreaks on The Netherlands population, 2009, using disability-adjusted life years. Epidemiol Infect. 2014;142(11):2412–21.View ArticlePubMedGoogle Scholar
- Smith RD. Responding to global infectious disease outbreaks: lessons from SARS on the role of risk perception, communication and management. Soc Science Med (1982). 2006;63(12):3113–23.View ArticleGoogle Scholar
- WHO Guidelines Approved by the Guidelines Review Committee. Pandemic Influenza Preparedness and Response: A WHO Guidance Document. Geneva: World Health Organization; 2009.Google Scholar
- McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–45.View ArticlePubMedGoogle Scholar
- Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United States? Milbank Q. 1998;76(4):517. –563, 509.View ArticlePubMedPubMed CentralGoogle Scholar
- Grol R, Wensing M. Implementatie Effectieve verbetering van de patiëntenzorg. Vierde, herziene druk edn. Amsterdam: Reed Business; 2011.Google Scholar
- Isken LD. Eindrapport evaluatie Q-koortsuitbraak in Noord-Brabant 2007. 2009.Google Scholar
- Timen A, Hulscher ME, Vos D, van de Laar MJ, Fenton KA, Van Steenbergen JE, et al. Control measures used during lymphogranuloma venereum outbreak, Europe. Emerg Infect Dis. 2008;14(4):573–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Timen A, Hulscher ME, Rust L, Van Steenbergen JE, Akkermans RP, Grol RP, et al. Barriers to implementing infection prevention and control guidelines during crises: experiences of health care professionals. Am J Infect Control. 2010;38(9):726–33.View ArticlePubMedGoogle Scholar
- Lawrence M, Olesen F. Indicators of Quality in Health Care. Eur J Gen Pract. 1997;3(3):103–8.View ArticleGoogle Scholar
- Torner N, Carnicer-Pont D, Castilla J, Cayla J, Godoy P, Dominguez A. Auditing the management of vaccine-preventable disease outbreaks: the need for a tool. PLoS One. 2011;6(1):e15699.View ArticlePubMedPubMed CentralGoogle Scholar
- Potter MA, Sweeney P, Iuliano AD, Allswede MP. Performance indicators for response to selected infectious disease outbreaks: a review of the published record. J Public Health Manag Pract. 2007;13(5):510–8.View ArticlePubMedGoogle Scholar
- Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476.View ArticlePubMedPubMed CentralGoogle Scholar
- Kotter T, Blozik E, Scherer M. Methods for the guideline-based development of quality indicators–a systematic review. Implement Sci. 2012;7:21.View ArticlePubMedPubMed CentralGoogle Scholar
- Fitch K, Bernstein SJ, Aquilar MD, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA Appropriateness Method user’s manual. Santa Monica, CA: The RAND Corporation; 2001.Google Scholar
- Gibson PJ, Theadore F, Jellison JB. The common ground preparedness framework: a comprehensive description of public health emergency preparedness. Am J Public Health. 2012;102(4):633–42.View ArticlePubMedPubMed CentralGoogle Scholar
- Van Ouwerkerk IMS, Isken LD, Timen A. Een kader voor het evalueren van infectieziekteuitbraken. Infectieziekte Bulletin. 2009;20(3):95–8.Google Scholar
- Cherifi S, Reynders M, Theunissen C. Hospital preparedness and clinical description of the 2009 influenza A (H1N1) pandemic in a Belgian tertiary hospital. J Hosp Infect. 2011;77(2):118–22.View ArticlePubMedGoogle Scholar
- Manigat R, Wallet F, Andre JC. From past to better public health programme planning for possible future global threats: case studies applied to infection control. Ann Ist Super Sanita. 2010;46(3):228–35.PubMedGoogle Scholar
- McVernon J, Mason K, Petrony S, Nathan P, LaMontagne AD, Bentley R, et al. Recommendations for and compliance with social restrictions during implementation of school closures in the early phase of the influenza A (H1N1) 2009 outbreak in Melbourne, Australia. BMC Infect Dis. 2011;11:257.View ArticlePubMedPubMed CentralGoogle Scholar
- Panda B, Stiller R, Panda A. Influenza vaccination during pregnancy and factors for lacking compliance with current CDC guidelines. J Matern Fetal Neonatal Med. 2011;24(3):402–6.View ArticlePubMedGoogle Scholar
- Inoue Y, Matsui K. Physicians’ recommendations to their patients concerning a novel pandemic vaccine: a cross-sectional survey of the 2009 influenza A/H1N1 pandemic in Japan. Environ Health Prev Med. 2011;16(5):320–6.View ArticlePubMedGoogle Scholar
- Sorbe A, Chazel M, Gay E, Haenni M, Madec JY, Hendrikx P. A simplified method of performance indicators development for epidemiological surveillance networks–application to the RESAPATH surveillance network. Rev Epidemiol Sante Publique. 2011;59(3):149–58.View ArticlePubMedGoogle Scholar
- Steelfisher GK, Blendon RJ, Bekheit MM, Mitchell EW, Williams J, Lubell K, et al. Novel pandemic A (H1N1) influenza vaccination among pregnant women: motivators and barriers. Am J Obstet Gynecol. 2011;204(6 Suppl 1):S116–23.View ArticlePubMedGoogle Scholar
- Dehecq JS, Baville M, Margueron T, Mussard R, Filleul L. The reemergence of the Chikungunya virus in Reunion Island on 2010: evolution of the mosquito control practices. Bull Soc Pathol Exot. 2011;104(2):153–60.View ArticlePubMedGoogle Scholar
- Mitchell T, Dee DL, Phares CR, Lipman HB, Gould LH, Kutty P, et al. Non-pharmaceutical interventions during an outbreak of 2009 pandemic influenza A (H1N1) virus infection at a large public university, April-May 2009. Clin Infect Dis. 2011;52 Suppl 1:S138–45.View ArticlePubMedGoogle Scholar
- Tsalik EL, Cunningham CK, Cunningham HM, Lopez-Marti MG, Sangvai DG, Purdy WK, et al. An Infection Control Program for a 2009 influenza A H1N1 outbreak in a university-based summer camp. J Am Coll Health. 2011;59(5):419–26.View ArticlePubMedGoogle Scholar
- Izadi M, Buckeridge DL. Optimizing the response to surveillance alerts in automated surveillance systems. Stat Med. 2011;30(5):442–54.PubMedGoogle Scholar
- Kavanagh AM, Bentley RJ, Mason KE, McVernon J, Petrony S, Fielding J, et al. Sources, perceived usefulness and understanding of information disseminated to families who entered home quarantine during the H1N1 pandemic in Victoria, Australia: a cross-sectional study. BMC Infect Dis. 2011;11:2.View ArticlePubMedPubMed CentralGoogle Scholar
- Schaade L, Reuss A, Haas W, Krause G. Pandemic preparedness planning. What did we learn from the influenza pandemic (H1N1) 2009? Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2010;53(12):1277–82.View ArticlePubMedGoogle Scholar
- Nicoll A. Pandemic risk prevention in European countries: role of the ECDC in preparing for pandemics. Development and experience with a national self-assessment procedure, 2005–2008. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2010;53(12):1267–75.View ArticlePubMedGoogle Scholar
- Lanari M, Capretti MG, Decembrino L, Natale F, Pedicino R, Pugni L, et al. Focus on pandemic 2009 H1N1 influenza (swine flu). Minerva Pediatr. 2010;62(3 Suppl 1):39–40.PubMedGoogle Scholar
- Ward KA, Armstrong P, McAnulty JM, Iwasenko JM, Dwyer DE. Outbreaks of pandemic (H1N1) 2009 and seasonal influenza A (H3N2) on cruise ship. Emerg Infect Dis. 2010;16(11):1731–7.View ArticlePubMedPubMed CentralGoogle Scholar
- Goodacre S, Challen K, Wilson R, Campbell M. Evaluation of triage methods used to select patients with suspected pandemic influenza for hospital admission: cohort study. Health Technol Assess (Winchester England). 2010;14(46):173–236.Google Scholar
- Martin R, Conseil A, Longstaff A, Kodo J, Siegert J, Duguet AM, et al. Pandemic influenza control in Europe and the constraints resulting from incoherent public health laws. BMC Public Health. 2010;10:532.View ArticlePubMedPubMed CentralGoogle Scholar
- Low CL, Chan PP, Cutter JL, Foong BH, James L, Ooi PL. International health regulations: lessons from the influenza pandemic in Singapore. Ann Acad Med Singap. 2010;39(4):325–8.PubMedGoogle Scholar
- Tay J, Ng YF, Cutter JL, James L. Influenza A (H1N1–2009) pandemic in Singapore–public health control measures implemented and lessons learnt. Ann Acad Med Singapore. 2010;39(4):313–24.PubMedGoogle Scholar
- Poalillo FE, Geiling J, Jimenez EJ. Healthcare personnel and nosocomial transmission of pandemic 2009 influenza. Crit Care Med. 2010;38(4 Suppl):e98–e102.View ArticlePubMedGoogle Scholar
- Vivancos R, Sundkvist T, Barker D, Burton J, Nair P. Effect of exclusion policy on the control of outbreaks of suspected viral gastroenteritis: Analysis of outbreak investigations in care homes. Am J Infect Control. 2010;38(2):139–43.View ArticlePubMedGoogle Scholar
- Harrison JP, Bukhari SZ, Harrison RA. Medical response planning for pandemic flu. Health Care Manag. 2010;29(1):11–21.Google Scholar
- Smith EC, Burkle Jr FM, Holman PF, Dunlop JM, Archer FL. Lessons from the front lines: the prehospital experience of the 2009 novel H1N1 outbreak in Victoria, Australia. Disaster Med Public Health Prep. 2009;3 Suppl 2:S154–9.View ArticlePubMedGoogle Scholar
- Bouye K, Truman BI, Hutchins S, Richard R, Brown C, Guillory JA, et al. Pandemic influenza preparedness and response among public-housing residents, single-parent families, and low-income populations. Am J Public Health. 2009;99 Suppl 2:S287–93.View ArticlePubMedPubMed CentralGoogle Scholar
- Seale H, McLaws ML, Heywood AE, Ward KF, Lowbridge CP, Van D, et al. The community’s attitude towards swine flu and pandemic influenza. Med J Aust. 2009;191(5):267–9.PubMedGoogle Scholar
- Rao S, Scattolini de Gier N, Caram LB, Frederick J, Moorefield M, Woods CW. Adherence to self-quarantine recommendations during an outbreak of norovirus infection. Infect Control Hosp Epidemiol. 2009;30(9):896–9.View ArticlePubMedGoogle Scholar
- O’Brien C, Selod S, Lamb KV. A national initiative to train long-term care staff for disaster response and recovery. J Public Health Manag Pract. 2009;15(2 Suppl):S20–4.View ArticlePubMedGoogle Scholar
- Davey VJ, Glass RJ, Min HJ, Beyeler WE, Glass LM. Effective, robust design of community mitigation for pandemic influenza: a systematic examination of proposed US guidance. PLoS One. 2008;3(7):e2606.View ArticlePubMedPubMed CentralGoogle Scholar
- Kivi M, Koedijk FD, van der Sande M, van de Laar MJ. Evaluation prompting transition from enhanced to routine surveillance of lymphogranuloma venereum (LGV) in the Netherlands. Euro surveillance. 2008;13(14):8087.PubMedGoogle Scholar
- Rasmussen SA, Jamieson DJ, Bresee JS. Pandemic influenza and pregnant women. Emerg Infect Dis. 2008;14(1):95–100.View ArticlePubMedPubMed CentralGoogle Scholar
- Cavey AM, Spector JM, Ehrhardt D, Kittle T, McNeill M, Greenough PG, et al. Mississippi’s infectious disease hotline: a surveillance and education model for future disasters. Prehosp Disaster Med. 2009;24(1):11–7.View ArticlePubMedGoogle Scholar
- Brandeau ML, Hutton DW, Owens DK, Bravata DM. Planning the bioterrorism response supply chain: learn and live. Am J Disaster Med. 2007;2(5):231–47.PubMedGoogle Scholar
- Burkle Jr FM, Hsu EB, Loehr M, Christian MD, Markenson D, Rubinson L, et al. Definition and functions of health unified command and emergency operations centers for large-scale bioevent disasters within the existing ICS. Disaster Med Public Health Prep. 2007;1(2):135–41.View ArticlePubMedGoogle Scholar
- Terhakopian A, Benedek DM. Hospital disaster preparedness: mental and behavioral health interventions for infectious disease outbreaks and bioterrorism incidents. Am J Disaster Med. 2007;2(1):43–50.PubMedGoogle Scholar
- Muto CA, Blank MK, Marsh JW, Vergis EN, O’Leary MM, Shutt KA, et al. Control of an outbreak of infection with the hypervirulent Clostridium difficile BI strain in a university hospital using a comprehensive “bundle” approach. Clin Infect Dis. 2007;45(10):1266–73.View ArticlePubMedGoogle Scholar
- Shigayeva A, Green K, Raboud JM, Henry B, Simor AE, Vearncombe M, et al. Factors associated with critical-care healthcare workers’ adherence to recommended barrier precautions during the Toronto severe acute respiratory syndrome outbreak. Infect Control Hosp Epidemiol. 2007;28(11):1275–83.View ArticlePubMedGoogle Scholar
- Spranger CB, Villegas D, Kazda MJ, Harris AM, Mathew S, Migala W. Assessment of physician preparedness and response capacity to bioterrorism or other public health emergency events in a major metropolitan area. Disaster Manag Response. 2007;5(3):82–6.View ArticlePubMedGoogle Scholar
- Rebmann T, English JF, Carrico R. Disaster preparedness lessons learned and future directions for education: results from focus groups conducted at the 2006 APIC Conference. Am J Infect Control. 2007;35(6):374–81.View ArticlePubMedGoogle Scholar
- Coombs GW, Van Gessel H, Pearson JC, Godsell MR, O’Brien FG, Christiansen KJ. Controlling a multicenter outbreak involving the New York/Japan methicillin-resistant Staphylococcus aureus clone. Infect Control Hosp Epidemiol. 2007;28(7):845–52.View ArticlePubMedGoogle Scholar
- Humphreys H. Control and prevention of healthcare-associated tuberculosis: the role of respiratory isolation and personal respiratory protection. J Hosp Infect. 2007;66(1):1–5.View ArticlePubMedGoogle Scholar
- Stone SP, Cooper BS, Kibbler CC, Cookson BD, Roberts JA, Medley GF, et al. The ORION statement: guidelines for transparent reporting of outbreak reports and intervention studies of nosocomial infection. Lancet Infect Dis. 2007;7(4):282–8.View ArticlePubMedGoogle Scholar
- Gupta RK, Zhao H, Cooke M, Harling R, Regan M, Bailey L, et al. Public health responses to influenza in care homes: a questionnaire-based study of local Health Protection Units. J Public Health (Oxf). 2007;29(1):88–90.View ArticleGoogle Scholar
- Vinson E. Managing bioterrorism mass casualties in an emergency department: lessons learned from a rural community hospital disaster drill. Disaster Manag Response. 2007;5(1):18–21.View ArticlePubMedGoogle Scholar
- Santos SL, Helmer DA, Fotiades J, Copeland L, Simon JD. Developing a bioterrorism preparedness campaign for veterans: Using focus groups to inform materials development. Health Promot Pract. 2007;8(1):31–40.View ArticlePubMedGoogle Scholar
- Uscher-Pines L, Bookbinder SH, Miro S, Burke T. From bioterrorism exercise to real-life public health crisis: lessons for emergency hotline operations. J Public Health Manag Pract. 2007;13(1):16–22.View ArticlePubMedGoogle Scholar
- Zerwekh T, McKnight J, Hupert N, Wattson D, Hendrickson L, Lane D. Mass medication modeling in response to public health emergencies: outcomes of a drive-thru exercise. J Public Health Manag Pract. 2007;13(1):7–15.View ArticlePubMedGoogle Scholar
- Murphy E, Black N, Lamping D, McKee C, Sanderson C. Consensus development methods, and their use in clinical guideline development: a review. Health Technol Assess. 1998;2(3):88.Google Scholar
- Campbell SM, Cantrill JA. Consensus methods in prescribing research. J Clin Pharm Ther. 2001;26(1):5–14.View ArticlePubMedGoogle Scholar
- Yeager VA, Menachemi N, McCormick LC, Ginter PM. The nature of the public health emergency preparedness literature 2000–2008: a quantitative analysis. J Public Health Manag Pract. 2010;16(5):441–9.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.