Diagnostic accuracy of in-house real-time PCR assay for Mycobacterium tuberculosis: a systematic review and meta-analysis

Background In recent years, studies on the diagnostic accuracy of in-house real-time PCR (hRT-PCR) assay for the detection of Mycobacterium tuberculosis (Mtb) have been reported with unignorable discrepancies. To assess the overall accuracy of the hRT-PCR assay for Mtb diagnosis in different samples for individuals with active pulmonary and extra-pulmonary Mtb infection, a systematic review and meta-analysis were performed. Methods The PUBMED, EMBASE, Web of Science, and Cochrane databases were searched up to June 2017 for eligible studies that estimated diagnostic sensitivity and specificity with the hRT-PCR assay in respiratory and non-respiratory samples in pulmonary and extra-pulmonary Mtb infection patients, with Mtb culture as the reference standard. Bivariate random effect models were used to provide pooled estimation of diagnostic accuracy. Further, subgroup and meta-regression analyses were performed to explore sources of heterogeneity. The risk of bias was assessed by the QUADAS-2 tool. Results Of the 3589 candidate studies, 18 eligible studies met our inclusion criteria. Compared to Mtb culture data, the pooled sensitivity and specificity were 0.96 and 0.92, respectively. The diagnostic odds ratio (DOR) was 192.96 (95% CI 68.46, 543.90), and the area under the summary ROC curve (AUC) was 0.9791. There was significant heterogeneity in sensitivity and specificity among the enrolled studies (p < 0.001). The studies with high-quality assessment and application of respiratory specimen were associated with better accuracy. Conclusions In low-income/high-burden settings, our results suggested that the hRT-PCR assay could be a useful test for the diagnosis of TB with high sensitivity and specificity. Electronic supplementary material The online version of this article (10.1186/s12879-019-4273-z) contains supplementary material, which is available to authorized users.


Background
Tuberculosis (TB) remains a major public health problem worldwide. In 2015, 10.4 million new cases of TB were reported. Approximately 1.4 million people died of the disease, and there were an additional 0.4 million deaths from co-infected with HIV [1]. Rapid diagnosis and treatment are pivotal for the effective control of TB in clinical practice [2]. Acid-fast staining and Mtb culture are classical Mtb diagnosis techniques. The acid-fast stain lacks sensitivity, and the culture requires several weeks for incubation [3,4]. The inherent limitations make it difficult for them to meet the requirement for early diagnostics [5].
Nucleic acid amplification tests (NAATs), such as polymerase chain reaction (PCR), which was developed in 1983, are now a common tool for the rapid diagnosis of many infectious diseases, including TB [6]. To date, some commercial tests, including COBAS TaqMan, Xpert MTB/RIF and the Abbott Real-Time MTB assay, have been used for TB diagnosis [7][8][9]. However, due to the limited infrastructure and medical resources, many clinical laboratories in suburban areas with high TB burden cannot afford these assays [1]. In-house polymerase chain reaction (hPCR) that uses equipment and reagents available from diverse suppliers in competitive markets may be more affordable, feasible, and sustainable than Xpert MTB/RIF [10]. hPCR is thus becoming popular in these areas. Several regions of the mycobacterial genome, such as IS6110 and 16S rDNA, have been used as targets for assays [11][12][13]. Meta-analysis of previous studies demonstrated that the diagnostic accuracy of hPCR assays was variable and inconsistent compared with commercial tests [14,15]. For example, the sensitivity of hPCR for tuberculosis meningitis varied between 0 and 100% [16]. In recent years, PCR technologies have improved markedly with the development of RT-PCR for the detection of mycobacterial infection [17]. This method has the advantage over conventional PCR in speed, automation, high sensitivity and specificity, and a low risk of cross-contamination [18,19]. In contrast to the commercial kits, an inexpensive RT-PCR would be particularly popular in regions that are short of medical equipment, such as Brazil, India, China, the Russian Federation, Southeast Asian, South Africa, and East Africa. Medical resources are limited in the majority of these countries [1]. Although recent studies have revealed that RT-PCR assays have good diagnostic performance for TB, there are unignorable discrepancies between their results [10,[20][21][22][23][24][25][26][27][28][29][30][31][32]. Moreover, none of the researchers could demonstrate precise diagnostic accuracy due to their limited statistical power. Therefore, by systematic review and meta-analysis, we explored factors associated with heterogeneity as well as diagnostic accuracy of the hRT-PCR assay for TB using data from previous studies.

Methods
The current meta-analysis was conducted according to the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [33]. Since the study was a systematic review and meta-analysis of published articles, patient consent or approval from the institutional ethics committee was not necessary.

Search strategy
We searched the following databases: PUBMED, EMBASE, Web of Science and Cochrane Library. All searches were up to date as of June 2017. The search terms used included "tuberculosis", "Mycobacterium tuberculosis", "nucleic acid amplification techniques", "real time PCR", "quantitative real-time polymerase chain reaction", "PCR, quantitative real-time", "quantitative realtime PCR", "real-time PCR, quantitative", "sensitivity and specificity", or "predictive value". In addition, the references of several previously published reviews on NAATs were searched for possible candidate articles.

Study selection
We included all available studies that reported the assessment of hRT-PCR assay for direct detection of TB.
Reasons for studies exclusion were (i) the reference standard was not culture proven Mtb; (ii) studies performed with other assays other than hRT-PCR assay (in mixed Methods Research, data were analysed for the eligible cases separately); (iii) application of hRT-PCR assay for determining drug resistance; (iv) incomplete data (lacking any of the availability data including truepositive, true-negative, false-positive and false-negative or these variables could not be calculated from the published data) were not extracted; (v) evaluation of hRT-PCR assay on animal specimens; and (vi) conference abstracts, letters, case reports, editorials, and reviews without original data were excluded.
Two investigators (LY and YHL) independently screened candidate literature by looking up the title and abstract. Then, the full texts of the potentially relevant articles were carefully read to determine whether they could be included. Disagreements were resolved by consensus between the two investigators.

Data extraction and quality assessment
Two investigators (LY and YHL) independently extracted accurate information from the ultimately included articles. Disagreements were resolved by consensus between the two investigators. The quality of the included studies was independently estimated by two investigators (HX and RG) using a Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2), which consists of seven domains [34]. A study was treated as a high-quality study when it had no domain with a high risk of bias and no domain with high applicability concerns.

Statistical analysis
Analysis was performed using Meta-Disc (version 1.4) software [35]. We pooled the data with the DerSimonian-Laird random effects model (REM), with the following pooled estimates: sensitivity, specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR-), and the diagnostic odds ratio (DOR).
Each study in the meta-analysis contributed a pair of numbers: sensitivity and specificity. A summary receiver operating characteristic (SROC) curve was constructed for the hRT-PCR assay [36]. A shoulder-like curve illustrates that the variability between studies may be due to the threshold effect. A non-shoulder-like curve indicates that sensitivity and specificity are not correlated. The overall diagnostic performance of that hRT-PCR assay was assessed as the area under the curve (AUC) (an AUC value of 100% indicates a perfect test, while an AUC of 50% signifies poor diagnostic accuracy) [37,38].
Heterogeneity between included studies refers to a high degree of variability in study results. The heterogeneity could be explained by variability in thresholds or differences in test methods and study characteristics. Chi-square and Fisher's exact tests were used to detect statistically significant heterogeneity. Heterogeneity between included studies was evaluated with subgroup (stratified) analysis and meta-regression analysis [39]. In the subgroup analysis, we computed pooled DOR estimates in various strata. The following factors as potential sources of heterogeneity: study design, target sequence, respiratory specimen versus non-respiratory specimen, the distribution of TB, and components of study quality.
The meta-regression model produces relative diagnostic odds ratios (RDOR) as the output [39]. An RDOR is a ratio of two DORs. An RDOR of 1.0 explains that a particular covariate does not affect the overall DOR. An RDOR> 1.0 explains that studies with a particular characteristic have a higher DOR than studies without this characteristic. For an RDOR< 1.0, the converse is true.
Finally, since publication bias is an important focus for meta-analyses of diagnostic studies [40], the potential publication bias of included studies was assessed by Deeks's funnel plot (Stata version 12.0; Stata Corp., College Station, TX).

Study search
Of the 3589 unique articles, we finally identified 15 eligible articles representing 18 independent studies (Fig. 1). The performance of the hRT-PCR assay in Mtb detection of clinical specimens was evaluated from all included studies with Mtb culture as a reference standard. Summary characteristics of the included studies are shown in Table 1. Eleven studies used respiratory specimens, and five used non-respiratory specimens. Two

Quality evaluation
We assessed the quality of the studies by QUADAS2. To assess the risk of bias regarding patient selection, three studies were deemed to be of case-control design, which compared diagnosed TB patients to non-TB individuals. Therefore, there was a high risk of bias based on the patient selection method after QUADAS2 assessment. As for "index test" evaluation, four studies failed to illustrate the blind working flow. Given the advanced acknowledgement of the reference test results, the bias could arise from the interpretation of the index test. These studies had an unclear risk of bias on the index test. For the reference standard, two studies did not provide sufficient description concerning the reference test results. Therefore, these studies were considered to have an unclear risk of bias on the reference test. No other domain had a high risk of bias or a high applicability concern (Fig. 2).

Diagnostic accuracy of hRT-PCR assay
When all 18 studies using the hRT-PCR assay were evaluated together, the overall sensitivity and specificity estimates were 0.96 (95% CI 0.95, 0.96) and 0.92 (95% CI 0.90, 0.93), respectively. The sensitivity and specificity of all studies are shown in the forest plot (Fig. 3a, b) Table 2). The overall accuracy of the hRT-PCR assay in a summary receiver operating characteristic (SROC) curve is displayed in Fig. 4, and the curve displayed a trade-off between sensitivity and specificity. The area under the SROC curve (AUC) was 0.9791, indicating a highly diagnostic accuracy. Overall, significant heterogeneity in sensitivity and specificity deserves more attention in the clinical applications of the hRT-PCR assay in TB detection.

Exploration of heterogeneity
Heterogeneity is an important concern on diagnostic meta-analysis. The threshold effect, method differences and study characteristics may lead to the variability. The SROC curve with studies was weighted by their inverse variance, as shown in Fig. 4. The non-shoulder-like curve indicated no threshold effect in the current metaanalysis. Furthermore, the Spearman correlation coefficient was 0.147, and the p value was 0.562. It illustrated no threshold effect. Subgroup analysis was also used to explore other factors that were associated with heterogeneity by stratifying data into relatively more homogeneous strata. The DOR estimates of the study characteristics are compared in Table 3. The heterogeneity could be explained in some strata, including specimen type, the distribution of TB, and quality of studies. However, even after stratification, the heterogeneity remained in the evaluation of diagnostic accuracy. We further performed a meta-regression analysis to explain the variation after subgroup analysis. As shown in Table 4, the RDOR was established from the meta-regression analysis using the restricted maximum Fig. 2 Summary of methodological quality of studies according to the QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies-2) tool. High-quality study: a study that had no domain with a high risk of bias and no domain with high applicability concerns; medium/moderate-quality study: a study that had domain with an unclear risk of bias or domain with unclear applicability concerns; low-quality study: a study that had a domain with a high risk of bias and domain with high applicability concerns likelihood (REML) method to measure between-study variance. Studies with respiratory specimens produced RDOR values that were significantly higher than those used non-respiratory specimens or both specimens.
Studies with a high-quality level produced RDOR that were significantly higher than those with medium quality levels or low-quality levels. The distribution of TB displayed a slightly higher RDOR but no statistical Fig. 3 Forest plot of specificity and sensitivity estimates. a Forest plot of sensitivity estimates and 95% confidence intervals (CI). The point estimates of sensitivity from each study are shown as solid circles. Error bars are 95% CI. Circles are proportional to study size. The pooled estimate is denoted by the diamond at the bottom. b Forest plot of specificity estimates and 95% CI. The point estimates of specificity from each study are shown as solid circles. Error bars are 95% CI. Circles are proportional to study size. The pooled estimate is denoted by the diamond at the bottom significance in the final regression model. Study design and target sequence did not produce a significant RDOR, indicating that the use of any study design and target sequence did not substantially affect diagnostic accuracy. Therefore, specimen types and quality of studies may affect accuracy heterogeneity. Evaluation of the Deeks' (p = 0.11) test did not show evidence of publication bias. Furthermore, the funnel plot did not display the presence of asymmetry (Fig. 5).

Principal findings
We summarized the evidence on the accuracy of the hRT-PCR assay for the diagnosis of TB and performed a meta-regression analysis to explore factors involved in in-house RT-PCR assay performance. This meta-analysis included 18 independent studies with a total of 97% AUC, indicating that the hRT-PCR assay for TB detection was useful in rapidly identifying TB cases and that negative data guaranteed the certainty for ruling out active TB. Since there is significant performance heterogeneity in our recruited studies, subgroup and metaregression analysis indicated that the use of respiratory specimens and studies with high quality were associated with better diagnostic accuracy of hRT-PCR.

Clinical implications
Even though the meta-analysis shows the power in evaluating the overall diagnostic accuracy of hRT-PCR, more caution is necessary to determine clinical accuracy due to significant heterogeneity. Previous meta-analyses [14,16,41] did not fully interpret the cause of heterogeneity found in hPCR results across studies. Our results showed that respiratory specimens and high-quality design were associated with better diagnostic accuracy of the hRT-PCR assay, which was consistent with a recent meta-analysis of the Xpert MTB/RIF PCR assay for the diagnosis of extra-pulmonary TB. There was a performance difference in the specimen site, with low sensitivity in pleural fluid (37%) and cerebrospinal fluid samples (69%) [42]. This finding was not surprising given the   [14,16,42]. The "casecontrol" study design and the IS6110 targeted sequence for hPCR were associated with better accuracy based on previous empirical research and meta-analyses [14,[43][44][45]. Some researchers were concerned that the case-control study might overestimate the diagnostic accuracy since it samples patients from the extreme ends of the clinical spectrum (an ideal, "extreme contrast" setting). For example, the sensitivity of a test is evaluated in seriously diseased subjects, and the specificity in healthy individuals [46]. In our meta-analyses, laboratory factors (such as target sequence and amplification technique) weighted more on accuracy than study design features. The IS6110 gene was widely used for both pulmonary and extra-pulmonary TB diagnosis [13,47,48]. Due to its multiple copies in the genome of the Mtb complex, PCR might result in better sensitivity [14]. However, our data demonstrated that study design with IS6110 had little impact on diagnostic accuracy. This is possible because RT-PCR used in our enrolled studies carries better advanced technology compared to conventional PCR. RT-PCR uses built-in automated thermocyclers and fluorimeters to monitor PCR reactions in a single tube format in which the reaction processes rapidly and minimizes the risk of contamination from product carryover [49]. Therefore, RT-PCR can provide reliable and repeatable results.  The performance of hRT-PCR was heterogeneous across studies; some patients could have false-positive hRT-PCR results and others false negative. Accuracy is related to the standard/reference assay, TB culture. Reliability is based on clinical diagnosis of TB disease. However, not all recruited studies have evaluated their hRT-PCR according to these standards. Caution is highly necessary for the clinical implications and applicability of hRT-PCR. The combination with other clinical information, such as the disease history, family medical records, microscopy screening and histopathology data, is recommended in clinical practice.
Previous meta-analyses of nucleic acid amplification (NAA) test accuracy PCR technology is widely used in the diagnosis of infectious diseases. Multiple commercial and in-house NAA techniques have been developed for TB diagnosis [41,50]. Xpert MTB/RIF, approved by the WHO and the FDA, is a novel, rapid, automated, cartridge-based NAA test that can simultaneously detect TB and rifampicin resistance directly on untreated sputum [51]. In addition, Xpert MTB/RIF was recommended for the diagnosis of TB in some special subjects, such as children and patients with certain forms of extra-pulmonary TB. A systematic review showed that Xpert MTB/RIF offered an acceptable sensitivity (62%) and specificity (98%) for the diagnosis of pulmonary tuberculosis in children [52]. Compared to hRT-PCR, the main features of Xpert MTB/RIF are its ability to detect resistance to rifampicin with a simple procedure and high cost. Xpert MTB/RIF may be practical for middle/high income regions. In fact, the majority of low-and middle-income countries, particularly those with limited resources, smear microscopy was still used for TB diagnosis [52]. Therefore, the hRT-PCR assay might substitute Xpert MTB/RIF in a low-income setting where multi-drug resistant TB is not prevalent. Another practical assay that might meet the needs of urban areas is loop-mediated isothermal amplification (LAMP) with a commercialized LAMP kit (Loopamp MTBC) [53]. A previous systematic review concerning the LAMP assay included 27 studies [54]; 9 out of the 27 studies evaluated the Loopamp MTBC, and the other 18 evaluated in-house LAMP assays. The summary of sensitivity and specificity for Loopamp MTBC were 80.9 and 96.5%, versus 93.0 and 91.8% for in-house LAMP assays, respectively. LAMP seems inferior to the RT-PCR tests in our analysis. Considering their low cost and simplicity, LAMP assays might be accepted in countries with limited resources.

Limitations of the review
Our review had some limitations. First, only one study evaluated the diagnostic test accuracy of the hRT-PCR assay for smear status, and only two studies included HIV-positive patients. Therefore, we could not determine the effect of smear and HIV status on the accuracy of the hRT-PCR assay. Second, we only included published studies in English, and this could have caused bias in our conclusion. Third, despite the fact that the subgroup analysis and meta-regression analysis could explain part of the observed heterogeneity in accuracy estimates, considerable heterogeneity remained unexplained. Finally, although we searched as many sources as possible, some eligible studies may have been missed.

Conclusion
In conclusion, based on the meta-analysis using the bivariate model, the diagnostic accuracy of the hRT-PCR assay for TB detection was acceptable. Subgroup and meta-regression analyses were performed, and we found that the diagnostic characteristics were different, depending on the specimen type and quality of the studies. Thus, the hRT-PCR assay, a relatively inexpensive assay compared to other commercial kits, has potential practical value for diagnosing TB, especially in low-income/ high-burden settings, where infrastructures and medical resources are limited.