- Research article
- Open Access
- Open Peer Review
Identifying heterogeneity in the Hawthorne effect on hand hygiene observation: a cohort study of overtly and covertly observed results
BMC Infectious Diseasesvolume 18, Article number: 369 (2018)
Observation and feedback are core strategies of hand hygiene (HH) improvement. Direct overt observation is currently the gold standard method. Observation bias, also known as the Hawthorne effect, is a major disadvantage of this method. Our aim was to examine the variation of the Hawthorne effect on HH observation in different healthcare groups and settings.
A prospective cohort study was performed in a tertiary teaching hospital during a 15-month period. Up to 38 overt observers (82% nurses) and 93 covert observers (81% medical students) participated in HH observation. The HH events observed overtly were matched for occupation, department, observation time, and location with those observed covertly. The data of matched pairs were then analysed to detect possible Hawthorne effects on different variables.
A total of 31,522 HH opportunities were observed (4581 overtly, 26,941 covertly). There were 3047 matched pairs after 1:1 matching of overt and covert observations. The overall HH compliance was higher with overt observation than with covert observation (78% vs. 55%, p < 0.001). The Hawthorne effect was nearly three times larger in nurses (30 percentage points) than in physicians (11 percentage points) and was significantly greater in outpatient clinics (41 percentage points) than in intensive care units (11 percentage points). The magnitude of the Hawthorne effect varied among healthcare worker occupations and observation locations (p values both < 0.001) but not among departments, observation times, or HH indications.
Heterogeneity in the Hawthorne effect may influence the interpretation of overt observations and prevent the correct identification of target populations with poor HH compliance. Therefore, directly observed HH compliance may not be an adequate performance indicator for infection control.
Hand hygiene (HH) is an effective measure to prevent healthcare-associated infection [1,2,3,4]. Observation and feedback make up one of the five main strategies to improve HH . Direct observation remains the gold standard method of evaluating HH compliance [3, 4]. However, direct observation has several limitations, including the ability to observe only a small percentage of HH opportunities in real-world scenarios, the need for a large number of observers and a considerable amount of time, the lack of a standardized training/validation process, and observation bias [5, 6].
Observation bias, also known as the Hawthorne effect, plays a critical role in evaluating HH compliance. When the Hawthorne effect was estimated by the difference between direct overt and covert observation results, the magnitude of the effect on HH compliance ranged from 7 to 16 percentage points (PPs) before 2009 [7,8,9] to 30–34 PPs after Five Moments for Hand Hygiene became a standard in 2009 [10, 11]. Srigley et al. even noted a surprising 3-fold increase in HH compliance under direct observation vs under an electronic monitoring system (EMS) . Hagel et al. reported a similar outcome by using electronic handrub dispensers to record HH events and captured a 2.6-fold higher HH density when healthcare workers (HCWs) were under observation than when they were not (21 vs 8 HH events per hour) . McLaws and Kwok demonstrated direct observation rates were inflated by an average of 55–64 PPs (2.8–3.1 times higher) than automated surveillance rates by an EMS in a medical ward . The Hawthorne effect may alter HCWs’ usual behaviour and often leads to overestimation of HH compliance [5, 6, 9, 15]. At least three methods have been proposed to help avoid or decrease the Hawthorne effect besides the use of EMS. These include 1) letting HCWs habituate to the presence of observers, although there is still no standard method for clinical application [3, 6]; 2) using indirect methods to monitor HH compliance, such as monitoring consumption of alcohol handrubs ; and 3) covertly observing HH compliance [3, 15].
Few studies have provided a detailed description and subgroup analysis of the Hawthorne effect [9, 11]. Kohli et al. conducted an observational study comparing the observation results of three infection control members and one student intern in three inpatient care units and found that the Hawthorne effect was more pronounced in high-performing than low-performing units. The authors speculated that the staff in high-performing units take pride in their work and may wish to show their good performance to recognized observers . Kovacs-Litman and colleagues described a much lower difference between covertly and overtly observed compliance rates in physicians than in nurses (19 PPs vs 41 PPs, p < 0.0001) . However, the two studies had several study design flaws, including the presence of selection bias [9, 11], a small number of observers , only HH opportunities occurred before and after contact with the patient or patient’s environment being observed , and a relatively small number of HH opportunities . Since healthcare facilities around the world spend so much time and utilize so many people for HH compliance observation (mostly overt), further research to investigate the Hawthorne effect is necessary .
The main purpose of the present study is to explore the heterogeneity of the Hawthorne effect in different HCW subgroups and settings. We analysed covert and overt observation data in the same hospital during the same period after controlling for observation place, time, department, and occupation category. The study results may clarify the impact of the Hawthorne effect under various HH circumstances.
All HH opportunities in the study were observed in Kaohsiung Veterans General Hospital (KVGH) from October 2012 through December 2013. Located in Kaohsiung City, Taiwan, KVGH is a tertiary teaching hospital with 1408 beds and more than 3600 HCWs. KVGH has regarded HH as a core element of infection control for nearly two decades and has implemented HH improvement strategies according to WHO guidelines soon after their issuance in 2009. The HH promotion campaign led to a significant decrease in the rate of healthcare-associated infection – from 3.7 to 3.1% (p = 0.002) – and reductions in overall healthcare cost and average hospital stay . Direct overt observation and feedback have been routinely performed in KVGH since 2009 as described in the WHO technique manual for observers. The overall rate of HH compliance in 2011 was 73% .
The HH compliance of HCWs was both overtly and covertly observed in wards, intensive care units, and outpatient clinics. The study protocol was approved by the KVGH Institutional Review Board (VGHKS13-CT4–05). All covert observers signed informed consent forms. Overt observers were exempted from signing informed consent forms because overt observation of HH compliance has long been a routine infection control practice in this hospital.
Overt observers were recruited among HCWs at KVGH. HCWs who were interested in becoming HH compliance observers could register to receive training. A total of 38 overt observers were included in the study after receiving training and certification. Those 38 observers were assigned to specific areas in the hospital and asked to observe for at least one hour every month. The observers could freely decide their observation dates and times. The minimum length of each observation period was 20 min. The hospital administration periodically and publicly commended the overt observers for their service but provided no monetary compensation.
The covert observers were medical students who participated in a HH training project in 2012–2013. The project was part of a campaign to improve HH in medical students after we found that doctors had the lowest HH compliance of all HCWs. The project recruited 149 students, who, similar to their overt observer counterparts, completed the same training and underwent the same validity check. In all, 93 of these students became covert observers. Their observer duties were performed at the sites of their internship rotations in teaching hospitals across Taiwan. A previous study described detailed observation results in hospitals other than KVGH . In this study, only covert observations in KVGH were used for analysis.
Training and certification
Overt and covert observers received nearly identical HH instruction from the same instructors. The course comprised lectures on HH knowledge, handrubbing and handwashing techniques, observation skills, video watching, and group discussion. The main teaching materials were the guidelines, video, and technical reference manual issued by the WHO [3, 18, 19]. After receiving the training, the observers were certified by passing a written test and demonstrating observer proficiency in five health care scenarios filmed by the infection control team. The process of HH observation via video watching also served as the means to reach inter-observer agreement. Those observers who were correct ≥80% of the time were considered qualified. Arrangements were made for both overt and covert observers to join discussion meetings every 3 months after being qualified. At those meetings, the concepts of HH observation will be reinforced, and problems about observation will be discussed to reach a consensus.
Covert observation techniques
The only difference between the courses taught to overt and covert observers was that covert observers were also taught techniques of covert observation. We designed a unique coding system to aid participants in remembering the observation results correctly and to reduce recall bias . The numbers 1–5 were used to represent 5 HH indications, along with R for handrubbing, W for handwashing, and N for no HH action. For example, when a physician washed his or her hands with handrub between visiting 2 consecutive patients, observers would quietly memorize 14R. The observers were encouraged to record the codes as soon as possible after observing no more than 2–3 HH opportunities.
Observation and reporting
The locations of overt observation were assigned by the infection control team of KVGH, and the overt observers decided their observation times and durations. The covert observers were encouraged to monitor compliance during their daily routines, such as ward rounds, in order not to interfere with their internship learning or be detected by other HCWs. The observation data records were uploaded to a website that could be accessed only by qualified observers and KVGH infection control team members. The covert observers were compensated $0.30 for every HH opportunity observed and reported. Overt observers received no monetary rewards.
HH opportunities observed in the study period were analysed according to different variables using descriptive statistics. Thereafter, the overt and covert observations were matched for patient department, HCW occupation, observation location, and observation time using a 1:1 exposure and non-exposure matching method. We used the difference in the rate of HH compliance between overt and covert observers in these matched pairs as a measure of the Hawthorne effect. McNemar’s tests were used to analyse the differences between overt and covert observer data in each category. Generalized estimating equations were then used to test for differences between overt and covert observer data among categories within each variable. For example, for the variable of occupation, McNemar’s test was used to assess the differences in the compliance rates of overt and covert observation for nurses, physicians, caregivers and other HCWs, respectively; generalized estimating equations were performed to test for differences between overt and covert observer data among physicians, nurses, caregivers, and other HCWs.
Only matched pairs with the same number of HH indications regardless of observation method were included in the analysis for the variable “number of indications per hand hygiene opportunity”. Only the matched pairs with the same HH indication regardless of observation method were used to analyse the variable “hand hygiene indication”. All analyses were performed using SPSS 22.0 for Windows (IBM, Armonk, NY).
Overall hand hygiene compliance
A total of 131 observers participated in the study from October 2012 through December 2013, including 38 overt observers and 93 covert observers. The overt observers were mainly nurses (82%), followed by infection control team members (8%), physicians (5%), and research assistants (5%). The covert observers were medical students (81%) as well as students in the departments of physical therapy (15%), Chinese medicine (2%), and dentistry (1%).
During the study period, 31,522 HH opportunities were observed, including 4581 overtly observed opportunities and 26,941 covertly observed opportunities. Before matching, the overall HH compliance rate was 81% by overt observation and 59% by covert observation.
Matched analysis results
There were 3047 matched pairs after matching. Nurses were the most common HCW category represented in the matched pairs (69%). Most HH actions occurred in the ward (67%) and during the day (81%). The rate of overall HH compliance was higher by overt observation than by covert observation (78% vs. 55%, p < 0.001), with a difference of 24 PPs. The higher rate of HH compliance by overt observation than by covert observation (p < 0.001) was also true for most hospital departments, HCW occupations, observation locations, and observation times but not among caregivers (p = 0.30) or other HCWs (p = 0.18). Table 1 describes the results for the 3047 matched pairs in detail.
Differences between overt and covert observation results
The difference between overt and covert observation results was approximately three times larger in nurses than in physicians (30 PPs vs. 11 PPs) and nearly four times larger in outpatient clinics than in intensive care units (41 PPs vs 11 PPs). Further analysis revealed that the differences between overt and covert observation results were different among various HCW occupations and observation locations (p values both < 0.001) but not among various departments (p > 0.05) (Table 2).
This study comprehensively investigated the extent of the Hawthorne effect on HH compliance observation and revealed that the Hawthorne effect differed significantly for HCW occupations and observation locations but not for departments, observation times, or HH indications. In this study, several attempts were made to reduce potential bias. In order to minimize observer bias, both overt and covert observers received the same training and underwent the same certification process; moreover, a large group of observers were recruited to observe a massive number of HH opportunities. To minimize selection bias, we used matched analysis to balance 4 important variables between the overt and covert observation groups. The overall Hawthorne effect was comparable between our study (24 PPs) and previous studies of direct HH compliance observation (7–34 PPs) [7,8,9,10,11]. The study results may influence how we interpret direct observer audits of HH compliance in the future.
The heterogeneity of the Hawthorne effect on HH observation has several impacts on HH monitoring. First, Hawthorne effect heterogeneity should be taken into consideration when identifying risk factors for not washing one’s hands. The identification of risk factors for HH noncompliance was often based on analyses of overt observation data in previous studies [20, 21]. For example, HH compliance was considered lower in physicians than in nurses [7, 20]. Is this proposition true, or do the observations reflect a greater Hawthorne effect in nurses than physicians ? The answer may even vary among health care facilities. Second, some experts may take advantage of the Hawthorne effect on HH observation to improve HH compliance, that is, using the strategy of frequent HH observation to stimulate HCWs to wash their hands more often. Implementing the strategy in the scenario with a higher Hawthorne effect will maximize the efficiency of HH improvement programmes. Third, using HH compliance by overt observation as a performance indicator to compare different HCW occupations, wards, and hospitals is not adequate. The major purpose of overt observation should remain the provision of feedback to HCWs.
Our finding that the Hawthorne effect impacted physicians less than nurses (11 PPs vs. 30 PPs) was comparable to the finding of Kovacs-Litman and colleagues (19 PPs vs. 41 PPs) . Our study did not show the previously reported more pronounced Hawthorne effect on HH in higher-performance units and lack of a significant Hawthorne effect on HH in low-performance units . Kwok et al. noted that a socially cohesive ward is more likely than a socially isolated ward to adopt an intervention for HH improvement . This difference may partially explain why units with high HH performance, which are presumed to be more social cohesive, have a larger response to overt observation, and vice versa. However, in our study, HH compliance was higher and the Hawthorne effect on compliance was lower in intensive care units than in wards or the outpatient department; moreover, HH compliance was lowest and the Hawthorne effect was highest in the surgical department. We consider the association between HH performance and the Hawthorne effect to be undetermined. Further studies are needed to address this issue.
The use of EMS to monitor HH compliance has attracted increasing attention in recent years [12,13,14, 23, 24]. An interesting point is that the estimated Hawthorne effect is often larger when HH compliance from overt observation is compared to EMS measures (1.6 to 3.1 times higher) [12,13,14] than to covertly observed measures (7–34 PPs or 12–69% higher) [7,8,9,10,11], including this study (24 PPs or 43% higher). This difference may imply that the Hawthorne effect can only be reduced, rather than eliminated, by covert observation. As long as the HCWs are not alone, whether the persons nearby tend to observe HH or not, the Hawthorne effect exists. To go one step further, the EMS may also produce a Hawthorne effect to a certain extent if the HCWs are aware of the existence of EMS. The differences in results between various observation methods may cause another form of Hawthorne effect heterogeneity.
There are several limitations to the study. A major limitation is that the Hawthorne effect was measured by calculating the difference between overt and covert observation results, as in previous studies [7,8,9, 11]. Although the Hawthorne effect may contribute the most to the difference, selection and observer bias will certainly exist despite statistical correction. For example, the covert observers may tend to observe HH at busier times to see more HH opportunities in less time due to the monetary incentive. Busier times were associated with lower HH compliance; thus, a form of selection bias may be present. In addition, the professions of the observers in overt and covert observation were considerably different, and we did not analyse the impact of observer profession (e.g., nurse vs physician observers) on the Hawthorne effect in the study. Observer bias may occur as a result. We believe that an accurate measurement of the Hawthorne effect can be obtained only through well-designed controlled trials, for example, the same group of observers conducting both overt and covert observations in designated areas during the same period of time. Until perfect results are available, the results retrieved from a large cohort with careful stratification and matching may provide useful information. Second, this investigation is a single-centre study that was conducted in a hospital with good HH implementation. The impact of the Hawthorne effect may vary from hospital to hospital, and we recommend caution in applying the study results to other hospitals. Third, covert observation may not be generally acceptable in other hospitals or countries, which would prevent the generalization of the study method.
In this study, we demonstrated not only the existence of the Hawthorne effect but also a significant difference in the Hawthorne effect among different HCW occupations and observation locations. Due to the heterogeneity of the Hawthorne effect, overt observation should not be used as the sole method of comparing performance among different clinical settings, HCWs, or hospitals. Overt observation continues to be the main method of HH observation because it is easy to perform and provides more information for feedback than EMS provides. However, better methods of benchmarking performance are needed.
Electronic monitoring system
Kaohsiung Veterans General Hospital
World Health Organization
Larson E. A causal link between handwashing and risk of infection? Examination of the evidence. Infect Control. 1988;9:28–36. Epub 1988/01/01
Larson E. Skin hygiene and infection prevention: more of the same or different approaches? Clin Infect Dis. 1999;29:1287–94. Epub 1999/10/19
WHO Guidelines approved by the guidelines review committee. WHO Guidelines on hand hygiene in health care: first global patient safety challenge clean care is safer care. Geneva: World Health Organization; 2009.
Boyce JM, Pittet D. Healthcare Infection Control Practices Advisory Committee; HICPAC/SHEA/APIC/IDSA Hand Hygiene Task Force. Guideline for Hand Hygiene in Health-Care Settings. Recommendations of the Healthcare Infection Control Practices Advisory Committee and the HICPAC/SHEA/APIC/IDSA Hand Hygiene Task Force. Society for Healthcare Epidemiology of America/Association for Professionals in Infection Control/Infectious Diseases Society of America. MMWR Recomm Rep. 2002;51(RR-16):1–45. quiz CE1–4. Epub 2002/11/07
Boyce JM. Electronic monitoring in combination with direct observation as a means to significantly improve hand hygiene compliance. Am J Infect Control. 2017;45:528–35. Epub 2017/05/01
Gould DJ, Creedon S, Jeanes A, Drey NS, Chudleigh J, Moralejo D. Impact of observing hand hygiene in practice and research: a methodological reconsideration. J Hosp Infect. 2017;95:169–74. Epub 2016/09/17
Eckmanns T, Bessert J, Behnke M, Gastmeier P, Ruden H. Compliance with antiseptic hand rub use in intensive care units: the Hawthorne effect. Infect Control Hosp Epidemiol. 2006;27:931–4. Epub 2006/08/31
Maury E, Moussa N, Lakermi C, Barbut F, Offenstadt G. Compliance of health care workers to hand hygiene: awareness of being observed is important. Intensive Care Med. 2006;32:2088–9. Epub 2006/10/24
Kohli E, Ptak J, Smith R, Taylor E, Talbot EA, Kirkland KB. Variability in the Hawthorne effect with regard to hand hygiene performance in high- and low-performing inpatient care units. Infect Control Hosp Epidemiol. 2009;30:222–5. Epub 2009/02/10
Pan SC, Tien KL, Hung IC, Lin YJ, Sheng WH, Wang MJ, et al. Compliance of health care workers with hand hygiene practices: independent advantages of overt and covert observers. PLoS One. 2013;8:e53746. Epub 2013/01/24
Kovacs-Litman A, Wong K, Shojania KG, Callery S, Vearncombe M, Leis JA. Do physicians clean their hands? Insights from a covert observational study. J Hosp Med. 2016;11:862–4. Epub 2016/07/06
Srigley JA, Furness CD, Baker GR, Gardam M. Quantification of the Hawthorne effect in hand hygiene compliance monitoring using an electronic monitoring system: a retrospective cohort study. BMJ Qual Saf. 2014;23:974–80. Epub 2014/07/09
Hagel S, Reischke J, Kesselmeier M, Winning J, Gastmeier P, Brunkhorst FM, et al. Quantifying the Hawthorne effect in hand hygiene compliance through comparing direct observation with automated hand hygiene monitoring. Infect Control Hosp Epidemiol. 2015;36:957–62. Epub 2015/04/24
McLaws ML, Kwok YLA. Hand hygiene compliance rates: fact or fiction? Am J Infect Control. 2018. https://doi.org/10.1016/j.ajic.2018.03.030. [Epub ahead of print].
Chen LF, Vander Weg MW, Hofmann DA, Reisinger HS. The Hawthorne effect in infection prevention and epidemiology. Infect Control Hosp Epidemiol. 2015;36:1444–50. Epub 2015/09/19
Chen JK, Wu KS, Lee SS, Lin HS, Tsai HC, Li CH, et al. Impact of implementation of the World Health Organization multimodal hand hygiene improvement strategy in a teaching hospital in Taiwan. Am J Infect Control. 2016;44:222–7. Epub 2015/12/24
Wu KS, Chen YS, Lin HS, Hsieh EL, Chen JK, Tsai HC, et al. A nationwide covert observation study using a novel method for hand hygiene compliance in health care. Am J Infect Control. 2017;45:240–4. Epub 2016/11/14
WHO offical website: Hand HygieneTraining Films. http://www.who.int/gpsc/5may/tools/training_education/en/ Accessed 10 Nov 2017.
WHO. Hand Hygiene Technical Reference Manual. Geneva: World Health Organization; 2009.
Mahfouz AA, El Gamal MN, Al-Azraqi TA. Hand hygiene non-compliance among intensive care unit health care workers in Aseer central hospital, South-Western Saudi Arabia. Int J Infect Dis. 2013;17:e729–32. Epub 2013/04/23
Korniewicz DM, El-Masri M. Exploring the factors associated with hand hygiene compliance of nurses during routine clinical practice. Appl Nurs Res. 2010;23:86–90. Epub 2010/04/28
Kwok YL, Harris P, McLaws ML. Social cohesion: the missing factor required for a successful hand hygiene program. Am J Infect Control. 2017;45:222–7. Epub 2016/12/9
Cheng VC, Tai JW, Ho SK, Chan JF, Hung KN, Ho PL, et al. Introduction of an electronic monitoring system for monitoring compliance with moments 1 and 4 of the WHO "my 5 moments for hand hygiene" methodology. BMC Infect Dis. 2011;11:151. https://doi.org/10.1186/1471-2334-11-151.
KPong S, Holliday P, Fernie G. Effect of electronic real-time prompting on hand hygiene behaviors in health care workers. Am J Infect Control. 2018;46:768–74. Epub 2018/3/2
This work was supported by the Taiwan Center for Disease Control (Grant No. DOH9X -DC-1208). Taiwan Center for Disease Control was not involved in the study design, study conduction, data analysis, and manuscript preparation.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Ethics approval and consent to participate
The study protocol was approved by the Kaohsiung Veterans General Hospital Institutional Review Board (VGHKS13-CT4-05). All covert observers signed informed consent forms. Overt observers were exempted from signing informed consent forms because overt observation of hand hygiene has long been a routine infection control practice in Kaohsiung Veterans General Hospital.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.