Framework of behavioral indicators for outcome evaluation of TB health promotion: a Delphi study of TB suspects and Tb patients

Selection of Delphi experts

A total of 17 TB control experts from national and provincial level TB control facilities participated in the Delphi process. During recruitment, potential experts were approached (initially via e-mail and later by telephone) and provided with detailed explanation of the study and its objectives. They were then asked if they would be interested in volunteering to participate. Those who expressed interest were asked to read the informed consent form, and were assured of confidentiality. They were sent an informed consent form to review, sign and return in an addressed stamped envelope as a conformation of their voluntary participation in the study. Ethical approval for the study was obtained from the Health Research Ethics Board of Third Military Medical University, China.

Instrument

The process of development of the main Delphi survey questionnaire is presented in Figure 1 (steps 1-3). First, we developed a detailed framework that was created based on an in-depth review of TB experts, information from previous studies [11, 47–54], and review of relevant literature on TB control and TB health promotion [17, 28, 55]. Second, TB health professionals working in TB dispensaries were asked to comment on the draft instrument, and to add as appropriate, other indicators and operational definitions. This resulted in two draft questionnaires (domains, subdomains, and indicators) on TB health promotion, one for TB suspects and one for TB patients. Third, the questionnaires were pre-tested on a convenient sample of 3 TB health workers who reviewed appropriateness and clarity of the questionnaire items. Fourth, as a result of pre-testing, proper phrasing was developed; respondents’ interpretations were evaluated, and Delphi questionnaires were finalized and applied as the survey instruments used to obtain responses from experts. The Delphi survey questionnaire that addressed health promotion indicators for TB suspects included 1 domain, 2 subdomains, and 3 indicators. The survey instrument for TB patients included 2 domains, 8 subdomains, and 14 indicators (Additional file 1: Table S1). Both instruments included operational definitions of all the potentially relevant indicators. Experts were asked to assess the importance and feasibility of each indicator on a 5-point Likert Scale in the following order and score: extremely unimportant/infeasible (1), unimportant/infeasible (3), somewhat important/feasible (5), important/feasible (7), and extremely important/feasible (9). Finally, the experts were given the options to delete, add, and modify indicators but were required to articulate the reasons behind their choices.

Information on experts’ authority included levels of familiarity with the indicators (extremely familiar, very familiar, somewhat familiar, somewhat unfamiliar, very unfamiliar, extremely unfamiliar) and judgment criteria on these indicators (theory analysis, practical experience, literature review, intuition) [56]. General information about the experts included age, position, professional title, education, work area, and years of experience in TB control.

Setting the consensus level

Procedures for the Delphi survey

Figure 1 shows the flow of the Delphi process (Step 4–7). In the first round, participants were asked to rank the importance and feasibility of the indicators to evaluate behavioral outcomes of individual level TB health promotion. The responses in the first-round survey were analyzed, using descriptive statistics, and the results were sent back to the experts for review and ratification. Items that achieved consensus of exclusion (i.e., > 70% of the experts scored the item ≤5) in the first round survey were excluded from the questionnaire for second round survey. Items for which were recommended for modification by the experts were revised and added to the second round survey; new items suggested by the experts were also added to the second round survey. In the second survey, participants were asked to re-rank the consensus results from the first round. Second Delphi responses that reached 70% consensus were determined as appropriate items (domains, subdomains and indicators) for assessing behavioral outcomes of individual level TB health promotion. The final framework was presented to experts for discussion and final consensus, leading to the final framework.

Statistical analysis

Median and mode were used to describe the central tendency of expert responses.

Coefficient of variation (CV) was used to describe the dispersals of expert responses. CV is the ratio of the standard deviation of the responses of the experts on a specific item to its corresponding mean (average). Therefore, the responses of the experts for each survey item in each round of the Delphi survey yielded one CV [70].

Characteristics of the experts

During the first round Delphi survey, questionnaires were sent to seventeen experts, sixteen of whom responded. The sixteen experts who participated in the survey were from thirteen provinces/regions representing North, South, West, and East China (Figure 2). All sixteen experts who responded to the first round survey also completed the second round survey. Descriptive information about the experts is presented in Table 1. All the experts had 8 or more years of experience (ranging from 8 to 36; mean = 11.9, SD = 5.1) as full-time professionals in TB control facilities. A majority (n = 12) had 11 to 30 years of experience.

Experts’ authority levels (Cr)

Results of round-1 survey

For the TB patients, medians and modes for the importance were 7 or 9 and CVs for the importance were lest 0.3 except for two items (“Behaviors related to deal with dishes”, and “Percentage of patients who used dishes and chopsticks separately”). However, the median for feasibility rankings of 6 items and mode for feasibility rankings of 5 items were 5; CVs for feasibility rankings of 12 items were more than 0.3 (Table 3). These results indicated that expert rankings had good central tendencies for most items in terms of importance, but not for the feasibility.

Indicator screening after first-round survey

Based on criteria for the consensus as earlier stated, indicators with a consensus score of >7 by >70% of the experts were considered as appropriate indicators and indicators with a consensus score of ≤5 by >70% of experts were excluded. Table 3 shows that for TB suspects, inclusion consensus was reached after round-1 survey on: one domain (“Health-care seeking behavior”), two sub-domains (“Care seeking behavior at onset of TB symptoms” and “Care-seeking pathways”) and two indicators (“Average number of health care provider encounters before diagnosis and percentage of patients who encountered ≥2 non-TB health providers” and “Percentage of patients with first non-TB health contact following onset of TB symptoms”).

For TB patients, consensus was reached on: one domain (“Adherence to treatment”), two sub-domains (“Adherence to their medication” and “Follow-up sputum microscopy”), and four indicators (“Percentage of patients who adhere to their medication”, “Percentage of patients who missed dose”, “Percentage of patients who kept follow-up sputum microscopy” and “Percentage of patients who often ventilated room”).

Regarding the indicators for TB suspects, experts did not suggest deleting any item; two indicators (“Period of patient delay and percentage of patients with longer patient delay” and “Average number of health care provider encounters before diagnosis and percentage of patients who encountered ≥2 health providers”) were suggested to be separated into two indicators for each; five new items (1 domain, 1 subdomain and 3 indicators) were suggested to be added to the questionnaire (Additional file 1: Table S4). Finally, 2 domains, 3 subdomains, and 8 indicators were included in the questionnaire for the second-round survey (Table 3).

As for indicators for TB patients, experts in the first-round survey suggested the deletion of: 4 subdomains (“Change unhealthy lifestyle”, “Wearing respirator in intensive phrase in public”, “Behaviors related to deal with dishes”, and “Isolation room”), and 2 indicators (“Percentage of patients who used dishes and chopsticks separately” and “Percentage of patients who had separate living room from others in household”). One item in the domain was modified (“Isolation behaviors” was changed into “Transmission prevention”). They also suggested the inclusion of eight new items to the questionnaire, including 1 domain, 4 sub-domains, and 3 indicators (Additional file 1: Table S4). Finally, 3 domains, 8 sub-domains, and 15 indicators were included in the questionnaire for the second-round survey (Table 3).

Results of second survey

Framework of indicators for behavioral outcome assessment of TB health promotion for TB suspects and patients

The following results are based on our criteria for inclusion and exclusion consensus. Regarding indicators for TB suspects, consensus was achieved for importance and feasibility except the feasibility of one domain (“Transmission prevention”, scored ≥7 by 68.8%) and one subdomain (“Interpersonal contact etiquette”, scored ≥7 by 56.3%) (Table 4). For TB patients, only one indicator (“Percentage of patients who reduced frequency of presence in public”) was scored ≥7 by <70% participants for both importance and feasibility (68.8% and 62.6%). Consensus was achieved on importance of the rest items and on feasibility of most items except for one domain (“Healthy lifestyle”, scored ≥7 by 68.8%), one subdomain (“Interpersonal contact etiquette”, scored ≥7 56.3%), and one indicator (“Percentage of patients who wear respirators in public”, scored ≥7 62.5%) (Table 4).

Strengthens and limitations

This study fills an important void in efforts to control TB globally by suggesting benchmarks that TB control programs around the world, which could adopt in their assessment of outcomes of their efforts. In the face of scarcity of resources, and considering that every resource spent on TB health promotion has opportunity costs in other sectors of any country’s economy, it is important to determine whether TB health promotion efforts are making an impact. This is critical in determining if resources are being spent judiciously or whether the resource should be spent on alternative strategies of proven effectiveness.

The Delphi process adopted in this study employed strict quality control measures. Local and international TB control experts were consulted in the design and implementation of the Delphi surveys. The expert panel consisted of professionals from thirteen provinces of China and demonstrated a high authority coefficient. Most experts had leading positions within their institutions in addition to extensive experience in TB control. Due to the multiple feedback processes inherent in the Delphi process, the potential low response rates, and striving to maintain robust feedback can be a challenge [59]. However, the return rates for this study were high at 94.1% and 100% for round 1 and 2 surveys, respectively.

Although the experts achieved high consensus on the domains, sub-domains and indicators, it is important to note that the extent to which participants agree with each other does not necessarily mean that the “correct” answer has been found [56]. Thus, there is the danger of over reliance on the final results without acknowledging the influence of bias and other factors on validity and reliability [56]. Further validation of the identified indicators by other national and global TB control programs is therefore warranted. It is also important to note that as a qualitative research method, Delphi studies do not, and are not intended to, produce statistically significant results. Rather, the results represent a synthesis of the opinions of the particular group involved in the process [73]. Thus, findings from this Delphi survey represent an important starting point for debate and consensus on global indicators for assessing the effectiveness of TB behavioral health promotion programs. The involvement of more experts and TB control programs from other parts of the globe will increase validity and reliability of the results [58]. To enhance generalizability, a number of strategies can be used: focus groups can be integrated or comparisons can be made with sub-domain-validated data [56]. A quasi-experimental design can be used as a follow-up method to test the results of the Delphi process [61], and a “consensus conference” can be organized to discuss the validity of the Delphi results. Finally, as noted in Table 5, much of the evaluation data for the proposed framework of indicators would rely on reports from patients and suspects. Since self-reports are known to be subject to social desirability, it is important that evaluation strategies for TB health promotion include good quality control and mixed-method approaches that seek to reduce bias and increase data quality.