- Research article
- Open Access
- Open Peer Review
This article has Open Peer Review reports available.
Tuberculosis mortality: patient characteristics and causes
© Lin et al.; licensee BioMed Central Ltd. 2014
Received: 8 August 2013
Accepted: 30 December 2013
Published: 3 January 2014
In the antibiotic era, tuberculosis (TB) still causes a substantial number of mortalities. We aimed to identify the causes and risks of death among TB patients.
Medical records of mortality cases of culture-proven TB diagnosed during 2003–2007 were reviewed. All TB deaths were classified into 2 groups (TB-related and non-TB-related), based on the underlying cause of death.
During the study period, 2016 cases (male: 71.1%) of culture-proven TB were identified. The mean age was 59.3 (range: 0.3–96) years. The overall mortality rate was 12.3% (249 cases) and the mean age at death was 74 years; 17.3% (43 cases) of all TB deaths were TB-related. Most of the TB-related deaths occurred early (median survival: 20 days), and the patient died of septic shock. Malignancy, liver cirrhosis, renal failure, and miliary and pneumonic radiographic patterns were all independent predictors for all TB deaths. Cavitary, miliary and pneumonic radiographic patterns were all significant predictive factors for TB-related death. Extrapulmonary involvement and liver cirrhosis were also factors contributing to TB-related death.
The majority of TB deaths were ascribed to non-TB-related causes. Managing TB as well as underlying comorbidities in a multidisciplinary approach is essential to improve the outcome of patients in an aging population. However, the clinical manifestations of patients with TB-related death vary; many progressed to fulminant septic shock requiring timely recognition with prompt treatment to prevent early death.
Tuberculosis (TB) remains a serious public health issue worldwide. Even in the era of effective chemotherapy, TB still accounts for a substantial number of deaths annually. Early diagnosis is challenging, even in areas with abundant medical resources . In 2012, there were an estimated 12 million TB cases globally, including 8.6 million new cases, and 1.3 million fatal cases . The global case-fatality rates are reported to be between 7% and 35% , and risk factors for death may include non-infective comorbidities, human immunodeficiency virus (HIV) infection and multidrug-resistant TB (MDRTB) . Since the World Health Organization (WHO) defined TB deaths as the number of TB patients dying during treatment, irrespective of cause , most studies have used all-cause mortality as a surrogate marker of mortality attributable to TB [3, 6–8]. Nevertheless, knowing the actual underlying cause of death, especially whether it was TB-related or not, is valuable in monitoring TB control and may help in identifying effective interventions [9, 10].
Some studies have investigated the actual causes of death among TB patients [10–17], and most relied on vital statistics registration or death certificates [10, 11, 13–15, 17]. However, they may not completely reflect the actual causes of death because of reporting bias due to inaccurate certificates in the registration system and the imprecise design in large population-based surveys [18–21]. Moreover, these studies were conducted in areas with a high prevalence of either HIV infection or MDRTB [3, 10–13]. These data may not be applicable to the rest of the world, and to countries such as Taiwan with an intermediate TB burden (incidence: 58/100,000; mortality: 3.2/100,000), and a low prevalence of MDRTB (1%) and HIV infection (0.7%) among newly-diagnosed TB cases . Hence, we conducted this study to investigate the causes and risk factors for death among TB patients in Taiwan.
This study was conducted in a 2500-bed university-affiliated hospital in northern Taiwan, and was approved by the ethics committee of National Taiwan University Hospital. In the approved protocol, the requirement for informed consent was waived due to the retrospective nature of the study. We searched mycobacterial laboratory and histology databases from 2003 through 2007 for all patients with newly diagnosed, culture-proven TB. We reviewed their medical records and obtained the following data: demographic characteristics, comorbidities, indications for seeking medical help, symptoms, radiological appearance, bacteriological investigation, laboratory findings, HIV serology, length of hospital stay and outcome. Along with HIV serostatus, all patients were followed for acquired immunodeficiency syndrome (AIDS)-defined illness until the end of the study. According to the WHO definition, TB death was defined as all-cause mortality before completing anti-TB treatment. All of our patients were followed until completely treated, death, or until December 31, 2010 (end of the study).
All TB deaths were classified into 2 groups according to the underlying cause of death – TB-related or non-TB-related, rather than the mode of death, such as respiratory failure or septic shock. In order to find the risk factors contributing to TB-related death without a confounding element, the defined TB-related death had to fulfill the following 3 criteria: 1) microbiological or pathologic evidence of sole TB infection without other pathogens cultured from sterile body fluid or tissue aseptically collected; 2) agreement between the reviewing physician and the underlying cause of death recorded on the death certificate or medical records from the primary care physician; and 3) no other cause that was equally likely to result in mortality. Otherwise, the mortality was classified as non-TB-related death, and the underlying cause of death was further determined. The mode of death for TB-related death was also recorded and defined as: 1) “respiratory failure” that preceded shock (septic or non-septic), with acute lung injury (ALI) or acute respiratory distress syndrome (ARDS). ALI or ARDS was diagnosed based on the consensus of the American-European Conference ; 2) “septic shock” preceding respiratory failure with or without ALI/ARDS. Patients with septic shock were required to have systemic inflammatory response syndrome, documented or suspected infection, and persistent hypotension despite fluid resuscitation ; and 3) “others”.
Survival time was defined as the interval between the day the index culture was plated and the time of death. An interval longer than 14 days between ordering the index TB culture and commencing anti-TB treatment was defined as a delay in treatment.2 Multidrug resistance (MDR) was defined as resistance against at least isoniazid and rifampicin. Chest radiographs of patients with TB-related death were categorized into miliary, cavitary or pneumonic patterns. A miliary pattern was defined as diffuse millet-sized nodules, a cavitary pattern was defined as radiographic opacity with an internal area of lucency, and a pneumonic pattern was defined as consolidation or infiltrates resembling bacterial pneumonia. When the chest radiograph pattern did not meet the above definitions, the film was classified as “others”.
For each mortality case, 2 age- (within 2 years), sex- and site of TB- (pulmonary or extrapulmonary) matched controls were selected. If sufficient controls could not be obtained, the acceptable age range was widened to 5 years. Controls were randomly selected from the baseline cohort of culture-proven TB patients diagnosed between 2003 and 2007 who completed a standard course of anti-TB treatment.
Data were expressed as percentage of the group for categorical variables and mean ± standard deviation (SD) for continuous variables, and were compared using Pearson’s X2 test or Fisher’s exact test and the independent-samples t test, respectively. For cases of TB-related death and non-TB-related death, comparison with the control group was performed. Survival curves were generated using the Kaplan-Meier method for all TB patients. Cox’s proportional hazards analysis for all variables (age, sex, sputum acid-fast smear, extrapulmonary involvement, malignancy, diabetes mellitus, liver cirrhosis, renal failure, HIV/AIDS, MDRTB, radiographic patterns, and delayed treatment) was employed to identify independent prognostic factors for all TB deaths in the initial model. In the subpopulation analysis of all TB deaths, TB-related death was compared with the control group. Age, sex and site of TB were not specifically matched with controls in this second model. Statistical analysis was performed with SPSS software (version 13.0 SPSS, Inc., Chicago, IL, USA).
All-cause mortality among TB patients
Underlying cause of death and median survival days among 249 culture-proven tuberculosis (TB) mortality cases
Underlying cause of death
Median survival days (range)
COPD with respiratory failure
Characteristics of all TB deaths: TB-related death vs. non-TB-related death
Characteristics of those with TB-related death and non-TB-related death and controls
TB-related death (n = 43)
Non-TB-related death (n = 206)
Control (n = 498)
Age (years): mean
AFB smear-positive sputum (%)
TB pleurisy or peritonitis
Hemoglobin (g/dl): mean
No parenchymal lesion
Delay in treatment >14 days
Clinical profiles of patients with TB-related death
Clinical presentation of 43 patients with TB-related death
Initial tentative diagnosis
Spontaneous bacterial peritonitis
Meningitis, nature undetermined
Fever of unknown origin
Acute myocardial infarction
COPD with acute exacerbation
Laboratory findings, thoracic radiographic presentations and clinical course of 43 patients with TB-related death
No parenchymal lesion
Platelet (109/L) <140
Albumin (g/dL) <3.5
White blood cell (109/L) ≥9
Hemoglobin (g/dL) <10
No. (%) of samples that grew M. tuberculosis
Delay from first visit to ordering TB culture >3 days
Mortality within 14 days
Risk factors for all TB deaths and TB-related death
Factors contributing to all TB deaths and TB-related death using the Cox-proportional hazard model
All TB deaths
Miliary pattern on CXR
Pneumonic pattern on CXR
Pneumonic pattern on CXR
Milary pattern on CXR
Cavitary pattern on CXR
Our study has confirmed that there are a substantial number of deaths associated with TB, even in the era of effective anti-TB medication and advanced mycobacteriology laboratories. The all-cause mortality rate of TB patients was 12.4%, and this was mainly due to non-TB-related causes (82.7%). Malignancy, liver cirrhosis, renal failure, and miliary and pneumonic radiographic patterns predicted mortality in all TB deaths. Patients who died of TB progressed rapidly and 37.2% was not diagnosed ante-mortem.
Our study showed a TB-related mortality rate of 2.1%, which was comparable to studies in Canada [15, 25] and in metropolitan area in China . A study in the USA reported a TB-related case fatality rate of 0.1%, but more severe forms, such as those with extrapulmonary involvement or a poor performance status, were excluded from the study . TB-related mortality rates in some regions of the world were higher than those in our study. Those are regions with more indigenous groups, poor access to health care or a higher MDRTB percentage, such as South Africa (7%) , Australia (8.7%)  and Russia (5.9-6.8%) [12, 13]. We also found that 82.7% of the deaths were due to non-TB-related causes. This is consistent with previous studies that have shown aging and underlying comorbidities as risk factors for TB deaths in developed or developing countries [11, 17]. This percentage was 98% in North America , 86% in the Netherlands , 75% in Russia  and 50.5% in China . Interventions and team care to treat TB and coexisting diseases are needed to decrease overall mortality .
Those who died of TB had a short median survival (20 days). The median time from visit to ordering TB culture, however, was only 3 days, suggesting that physicians in our hospital were alert to TB. Yet more than one-third of those who died of TB did not receive anti-TB treatment before their death, probably because: 1) the patients often had negative sputum smears for acid-fast bacilli; 2) most patients whose diagnosis of TB was made postmortem (14/16, or 88%) had a rapidly fatal course (<14 days), and succumbed before culture results became available; 3) patients in the fatality group often showed a pneumonic pattern on chest radiograph, and were treated as having bacterial pneumonia; 4) they were often older patients with underlying comorbidity and a high risk of developing adverse drug effects, so physicians may not start empirical anti-TB treatment even when TB is highly suspected.
On the other hand, even among patients whose TB was diagnosed ante-mortem, the median time from treatment initiation to death (23 days) was short. This finding was similar to that of previous reports, in which many patients died of TB within a short period of time, ranging from 1 week to 3 months after starting treatment [7, 9, 13–15]. It was speculated that these patients were too ill on arrival and their outcomes could not be reversed even after treatment . The initial tentative diagnosis of our fatal TB patients was bacterial pneumonia in 39.5%, and sepsis or other severe infections (e.g. peritonitis, meningitis, etc.) in 20%, suggesting that TB should be included in differential diagnoses in patients presenting with severe pneumonia, sepsis or other severe infections such as meningitis (Table 3). Our study, similar to a previous report, failed to demonstrate that delayed treatment was an important factor for mortality . This finding does not minimize the importance of a prompt diagnosis and treatment of TB; rather the negative association is probably related to the retrospective design and definition of delayed treatment. Our observation that 40.7% of those who were diagnosed ante-mortem had received anti-TB treatment for less than 14 days before death suggested that some of the mortality may have been reversible if anti-TB therapy were started earlier.
Several comorbidities that are risk factors for all-cause mortality during anti-TB treatment have been noted. However the results were not consistent, probably due to the heterogeneous nature of the studied population. In a review by Waitt , comorbidities were risk factors for TB deaths in areas with high TB incidence and HIV prevalence, rather than in areas with low TB incidence and HIV prevalence. In our study, with TB and non-TB-related death defined separately, only liver cirrhosis was found to be a risk factor for TB-related death. Two case–control studies in the literature also reported that liver disease or hyperbilirubinemia was an independent risk factor for TB-related mortality [12, 14]. The association of liver disease and TB-related mortality has seldom been reported, and this could be due to the lack of availability of diagnostic facilities for liver disease in resource-poor regions. This also indicates that physicians in resource-limited areas should be proactive in treating TB patients with physical signs of liver disease.
Previously reported TB-related critical conditions were mainly miliary TB with respiratory failure and ARDS [26, 27]. In our observation, most (17/43, 39.5%) of the mortality cases presented with a pneumonic radiographic pattern and the patients died of septic shock rather than respiratory failure in a catastrophic course. Only 4 patients (9.3%) presented with miliary TB. Disseminated TB is difficult to diagnose if miliary lesions are not present on the radiograph . TB with either a miliary or diffuse pneumonic pattern can progress to septic shock with multiple organ failure. This is termed sepsis tuberculosa gravissima, a condition that was described well before 1951 , but seemed to be left unnoticed after the advent of effective chemotherapy. However, in TB-non-endemic areas, it has been reported in non-HIV patients since late 1990 [30–32]. These patients often presented with bilateral diffuse alveolar and interstitial infiltrates, septic shock and hypoxemic respiratory failure. The refractory septic shock often led to multi-organ failure and death in hours or days. Therefore, to prevent these deaths, a high index of suspicion for TB needs to be maintained and prompt treatment should be initiated in pneumonic patients at risk of TB. The presence of diffuse pneumonic infiltrates and the high yield of sputum culture for M. tuberculosis (69.8% in our study) also suggest that sputum and bronchoalveolar lavage fluid PCR for M. tuberculosis may help an early diagnosis. However, the low positive rate of acid-fast stain smear in our study suggested that we may need to consider other rapid adjunctive tests, such as a urine lipoarabinomannan test for the early diagnosis of TB or extrapulmonary TB in resource-limited areas .
Our study has some limitations. First, we did not have autopsy reports, which are often considered the “gold standard” for underlying cause of death. However, misclassification bias should have been minimized by the strict criteria defining TB-related death. Second, co-infection with other bacteria was classified as non-TB-related death, according to the inclusion criterion. This may have resulted in an underestimation of the numbers of TB-related deaths. However, the strict criteria were used to prevent confounding while analyzing TB-related death. Third, our study was done retrospectively in an inner-city hospital, and information regarding many socio-cultural factors, such as homelessness, drug or alcohol addiction, HIV infection, etc., may not have been available. Thus our results may not be applicable to areas where these pre-hospital factors are important. Finally, gathering information on more detailed clinical parameters, such as central venous pressure or central venous oxygen saturation, was not done due to the retrospective nature of the study design. Further prospective clinical studies are needed for a better understanding of TB deaths.
In the era of effective chemotherapy against TB, we found that a significant proportion of mortality among aging TB patients was due to underlying comorbidities. Liver cirrhosis was an important predictor for TB-related death. The majority of these patients presented as bacterial pneumonia and died rapidly due to septic shock. Our findings could help establish a better understanding of cause-specific TB mortality and help identify TB patients at risk.
We thank the TB controllers and laboratory staffs in National Taiwan University Hospital who collected and recorded the data used in this study.
- Smith A, Miller RF, Story A, Booth HL: A&E department: a missed opportunity for diagnosis of TB?. Thorax. 2006, 61 (4): 364-365. 10.1136/thx.2005.053637.View ArticlePubMedPubMed CentralGoogle Scholar
- World Health Organization: Global tuberculosis control: WHO report 2013. 2013, Geneva, Switzerland: World Health OrganizationGoogle Scholar
- Oursler KK, Moore RD, Bishai WR, Harrington SM, Pope DS, Chaisson RE: Survival of patients with pulmonary tuberculosis: clinical and molecular epidemiologic factors. Clin Infect Dis. 2002, 34 (6): 752-759. 10.1086/338784.View ArticlePubMedGoogle Scholar
- Waitt CJ, Squire SB: A systematic review of risk factors for death in adults during and after tuberculosis treatment. Int J Tuberc Lung Dis. 2011, 15 (7): 871-885. 10.5588/ijtld.10.0352.View ArticlePubMedGoogle Scholar
- World Health Organization: Global Tuberculosis Programme. A framework for effective tuberculosis control. WHO/TB/94.179. 1994, Geneva, Switzerland: WHOGoogle Scholar
- Rao VK, Iademarco EP, Fraser VJ, Kollef MH: The impact of comorbidity on mortality following in-hospital diagnosis of tuberculosis. Chest. 1998, 114 (5): 1244-1252. 10.1378/chest.114.5.1244.View ArticlePubMedGoogle Scholar
- Borgdorff MW, Veen J, Kalisvaart NA, Nagelkerke N: Mortality among tuberculosis patients in The Netherlands in the period 1993–1995. Eur Respir J. 1998, 11 (4): 816-820. 10.1183/09031936.98.11040816.View ArticlePubMedGoogle Scholar
- Hansel NN, Merriman B, Haponik EF, Diette GB: Hospitalizations for tuberculosis in the United States in 2000: predictors of in-hospital mortality. Chest. 2004, 126 (4): 1079-1086. 10.1378/chest.126.4.1079.View ArticlePubMedGoogle Scholar
- Nisar M, Davies PD: Current trends in tuberculosis mortality in England and Wales. Thorax. 1991, 46 (6): 438-440. 10.1136/thx.46.6.438.View ArticlePubMedPubMed CentralGoogle Scholar
- Moorman J, Edginton ME: Cause of death of patients on treatment for tuberculosis: a study in a rural South African hospital. Int J Tuberc Lung Dis. 1999, 3 (9): 786-790.PubMedGoogle Scholar
- Shen X, Deriemer K, Yuan Z, Shen M, Xia Z, Gui X, Wang L, Mei J: Deaths among tuberculosis cases in Shanghai, China: who is at risk?. BMC Infect Dis. 2009, 17 (9): 95-View ArticleGoogle Scholar
- Dewan PK, Arguin PM, Kiryanova H, Kondroshova NV, Khorosheva TM, Laserson K, Kluge H, Jakubowiak W, Wells C, Kazionny B: Risk factors for death during tuberculosis treatment in Orel Russia. Int J Tuberc Lung Dis. 2004, 8 (5): 598-602.PubMedGoogle Scholar
- Mathew TA, Ovsyanikova TN, Shin SS, Gelmanova I, Balbuena DA, Atwood S, Peremitin GG, Strelis AK, Murray MB: Causes of death during tuberculosis treatment in Tomsk Oblast Russia. Int J Tuberc Lung Dis. 2006, 10 (8): 857-863.PubMedGoogle Scholar
- Walpola HC, Siskind V, Patel AM, Konstantinos A, Derhy P: Tuberculosis-related deaths in Queensland, Australia, 1989–1998: characteristics and risk factors. Int J Tuberc Lung Dis. 2003, 7 (8): 742-750.PubMedGoogle Scholar
- Xie HJ, Enarson DA, Chao CW, Allen EA, Grzybowski S: Deaths in tuberculosis patients in British Columbia, 1980–1984. Tuber Lung Dis. 1992, 73 (2): 77-82. 10.1016/0962-8479(92)90059-S.View ArticlePubMedGoogle Scholar
- Zafran N, Heldal E, Pavlovic S, Vuckovic D, Boe J: Why do our patients die of active tuberculosis in the era of effective therapy?. Tuber Lung Dis. 1994, 75 (5): 329-333. 10.1016/0962-8479(94)90077-9.View ArticlePubMedGoogle Scholar
- Sterling TR, Zhao Z, Khan A, Chaisson RE, Schluger N, Mangura B, Weiner M, Vernon A: Mortality in a large tuberculosis treatment trial: modifiable and non-modifiable risk factors. Int J Tuberc Lung Dis. 2006, 10 (5): 542-549.PubMedGoogle Scholar
- Heldal E, Naalsund A, Kongerud J, Tverdal A, Boe J: Deaths from active tuberculosis: can we rely on notification and mortality figures?. Tuber Lung Dis. 1996, 77 (3): 215-219. 10.1016/S0962-8479(96)90003-9.View ArticlePubMedGoogle Scholar
- Hooi LN, Goh KY: A hospital based audit of tuberculosis deaths. Med J Malaysia. 1995, 50 (4): 306-313.PubMedGoogle Scholar
- Dye C, Bassili A, Bierrenbach AL, Broekmans JF, Chadha VK, Glaziou P, Gopi PG, Hosseini M, Kim SJ, Manissero D, et al: Measuring tuberculosis burden, trends, and the impact of control programmes. Lancet Infect Dis. 2008, 8 (4): 233-243. 10.1016/S1473-3099(07)70291-8.View ArticlePubMedGoogle Scholar
- Kircher T, Nelson J, Burdo H: The autopsy as a measure of accuracy of the death certificate. N Engl J Med. 1985, 313 (20): 1263-1269. 10.1056/NEJM198511143132005.View ArticlePubMedGoogle Scholar
- Center for Disease Control, Department of Health, Executive Yuan, Taiwan: Taiwan Tuberculosis Control Report 2010. Available at: http://www.cdc.gov.tw
- Bernard GR, Artigas A, Brigham KL, Carlet J, Falke K, Hudson L, Lamy M, Legall JR, Morris A, Spragg R: The American-European consensus conference on ARDS: definitions, mechanisms, relevant outcomes, and clinical trial coordination. Am J Respir Crit Care Med. 1994, 149 (3 Pt 1): 818-824.View ArticlePubMedGoogle Scholar
- Dellinger RP, Levy MM, Carlet JM, Bion J, Parker MM, Jaeschke R, Reinhart K, Angus DC, Brun-Buisson C, Beale R, et al: Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2008. Crit Care Med. 2008, 36 (1): 296-327. 10.1097/01.CCM.0000298158.12101.41.View ArticlePubMedGoogle Scholar
- Moniruzzaman A, Elwood RK, Wong H, Kazanjian A, FitzGerald JM: A seventeen-year study of TB diagnosed postmortem in British Columbia, Canada. Am J Respir Crit Care Med. 2009, 179: A5279-Google Scholar
- Dyer RA, Chappell WA, Potgieter PD: Adult respiratory distress syndrome associated with miliary tuberculosis. Crit Care Med. 1985, 13 (1): 12-15. 10.1097/00003246-198501000-00004.View ArticlePubMedGoogle Scholar
- Lintin SN, Isaac PA: Miliary tuberculosis presenting as adult respiratory distress syndrome. Intensive Care Med. 1988, 14 (6): 672-674.View ArticlePubMedGoogle Scholar
- Kim YJ, Pack KM, Jeong E, Na JO, Oh YM, Lee SD, Kim WS, Kim DS, Kim WD, Shim TS: Pulmonary tuberculosis with acute respiratory failure. Eur Respir J. 2008, 32 (6): 1625-1630. 10.1183/09031936.00070907.View ArticlePubMedGoogle Scholar
- Pagel W: The evolution of tuberculosis in man. Pulmonary Tuberculosis. Edited by: Pagel W, Simmonds FAH, Norman M. 1964, London: Oxford Medical Publisher, pp 147-pp 161. 4Google Scholar
- Angoulvant D, Mohammedi I, Duperret S, Bouletreau P: Septic shock caused by Mycobacterium tuberculosis in an non-HIV patient. Intensive Care Med. 1999, 25 (2): 238-10.1007/s001340050825.View ArticlePubMedGoogle Scholar
- Pene F, Papo T, Burdy-Gulphe L, Cariou A, Piette JC, Vinsonneau C: Septic shock and thrombotic microangiopathy due to Mycobacterium tuberculosis in a nonimmunocompromised patient. Arch Intern Med. 2001, 161 (10): 1347-1348. 10.1001/archinte.161.10.1347.View ArticlePubMedGoogle Scholar
- Michel P, Barbier C, Loubiere Y, Hayon JH, Ricôme JL: Three cases of septic shock due to tuberculosis without HIV pathology. Intensive Care Med. 2002, 28 (12): 1827-1828. 10.1007/s00134-002-1526-9.View ArticlePubMedGoogle Scholar
- Achkar JM, Lawn SD, Moosa MY, Wright CA, Kasprowicz VO: Adjunctive tests for diagnosis of tuberculosis: serology, ELISPOT for site-specific lymphocytes, urinary lipoarabinomannan, string test, and fine needle aspiration. J Infect Dis. 2011, 15 (Suppl 4): S1130-S1141.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2334/14/5/prepub
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.