Episodic disability questionnaire (EDQ) measurement properties among adults living with HIV in Canada, Ireland, United Kingdom, and United States

Abstract

Background

The Episodic Disability Questionnaire (EDQ) is a generic 35-item patient-reported outcome measure of presence, severity and episodic nature of disability. We assessed the measurement properties of the Episodic Disability Questionnaire (EDQ) with adults living with HIV.

Methods

We conducted a measurement study with adults living with HIV in eight clinical settings in Canada, Ireland, United Kingdom, and United States. We electronically administered the EDQ followed by three reference measures (World Health Organization Disability Assessment Schedule; Patient Health Questionnaire; Social Support Scale) and a demographic questionnaire. We administered the EDQ only 1 week later. We assessed the internal consistency reliability (Cronbach’s alpha; > 0.7 acceptable), and test–retest reliability (Intra Class Correlation Coefficient; > 0.7 acceptable). We estimated required change in EDQ domain scores to be 95% certain that a change was not due to measurement error (Minimum Detectable Change (MDC95%)). We evaluated construct validity by assessing 36 primary hypotheses of relationships between EDQ scores and scores on the reference measures (> 75% hypotheses confirmed indicated validity).

Results

Three hundred fifty nine participants completed the questionnaires at time point 1, of which 321 (89%) completed the EDQ approximately 1 week later. Cronbach’s alpha for internal consistency ranged from 0.84 (social domain) to 0.91 (day domain) for the EDQ severity scale, and 0.72 (uncertainty domain) to 0.88 (day domain) for the EDQ presence scale, and 0.87 (physical, cognitive, mental-emotional domains) to 0.89 (uncertainty domain) for the EDQ episodic scale. ICCs for test–retest reliability ranged from 0.79 (physical domain) to 0.88 (day domain) for the EDQ severity scale and from 0.71 (uncertainty domain) to 0.85 (day domain) for the EDQ presence scale. Highest precision was demonstrated in the severity scale for each domain (MDC95% range: 19–25 out of 100), followed by the presence (MDC95% range: 37–54) and episodic scales (MDC95% range:44–76). Twenty-nine of 36 (81%) construct validity hypotheses were confirmed.

Conclusions

The EDQ possesses internal consistency reliability, construct validity, and test–retest reliability, with limited precision when administered electronically with adults living with HIV across in clinical settings in four countries. Given the measurement properties, the EDQ can be used for group level comparisons for research and program evaluation in adults living with HIV.

Background

People with Human Immunodeficiency Virus (HIV) are living longer and may be aging with health-related challenges related to other health conditions in addition to HIV, often referred to as disability [1,2,3]. Disability is defined as any health related challenge, where health challenges may persist, or fluctuate on a daily basis or over the longer course of living with HIV [4, 5].

Measuring disability is important for identifying and determining the impact of health challenges, improving communication between providers and patients, and evaluating the effect of interventions [6,7,8]. We developed the Episodic Disability Questionnaire (EDQ) a patient-reported outcome measure of disability, derived from an HIV-specific measure of disability, the HIV Disability Questionnaire (HDQ) developed to describe disability experienced by adults living with HIV [9]. The EDQ is based on a conceptual framework of disability [4, 10], and the foundational HDQ, which possessed validity, reliability, and sensibility for use among adults living with HIV in Canada, Ireland, the United States (US), and United Kingdom (UK) [11,12,13,14,15]. However, at 69 items, the HDQ was lengthy posing challenges for clinical use [16].

We shortened the HDQ to a 35-item version Short-Form HIV Disability Questionnaire (SF-HDQ), spanning six domains: i) physical; ii) cognitive; iii) mental-emotional health challenges, iv) uncertainty or worrying about the future, v) difficulties carrying out day-to-day activities, and vi) challenges to social inclusion [17]. The SF-HDQ possessed sensibility and utility for use with adults living with HIV in clinical and community-based settings in Canada, Ireland, US, and UK [18]. However, adults living with HIV and HIV health care practitioners questioned the need to specify HIV as the source of health challenges for some SF-HDQ items related to uncertainty in finance and housing, and social inclusion [18]. Attributing specific health challenges to HIV may be difficult for individuals, as challenges may not be directly due to HIV, but rather attributed to side effects from treatments, aging or concurrent health conditions [18]. Hence, we removed the HIV specificity of items and instructions in the SF-HDQ to establish a new generic measure of episodic disability, renamed the EDQ. The EDQ has the potential to describe the multi-dimensional and episodic nature of disability, regardless of its source, broadening the potential applicability for use with other health conditions. Despite a similar domain structure, the psychometric properties of the EDQ, when administered electronically across different clinical contexts, with adults living with HIV were unknown.

Our aim was to assess the measurement properties of the newly established EDQ for its ability to measure the presence, severity and episodic nature of disability among adults living with HIV in Canada, Ireland, United Kingdom, and United States.

Methods

We conducted a cross-sectional measurement study involving administration of the EDQ and criterion measures with adults living with HIV in Canada, United Kingdom, Ireland and United States. We followed the (COnsensus‐based Standards for the selection of health Measurement Instruments (COSMIN) guidelines for assessment and reporting of psychometric properties of the EDQ [19,20,21,22,23].

Study setting

This study was conducted at eight clinical settings, in five cities, in four countries: Canada (Casey House, Toronto), Ireland (Department of Genitourinary Medicine and Infectious Diseases (GUIDE), St. James’s Hospital, Dublin), the United States (The University of Colorado Infectious Diseases Group Practice Clinic, University of Colorado, Denver), and United Kingdom (Royal Sussex County Hospital, Brighton, and Chelsea and Westminster NHS Foundation Trust, London involving four sites: Kobler Rehabilitation Clinic, 10 Hammersmith Broadway, 56 Dean Street, and West Middlesex University Hospital). The Dublin, Denver, and UK sites are HIV outpatient clinics, and the Toronto site is a specialty HIV hospital including an inpatient and day health program for people living with and at risk of HIV.

We received ethics approval from the University of Toronto (Protocol # 38152), University of Colorado (Protocol # 19–1895), St. James’s Hospital (Protocol # 2019–12), London Fulham Research Ethics Committee (REC reference: 20/LO/0909) and NHS Health Research Authority (IRAS project ID: 284075).

Participants

We recruited adults (18 years of age or older) living with HIV from each site using a recruitment poster asking interested individuals to contact the local study investigator. Informed consent was obtained from the participants in the study by checking ‘yes I consent to participate in the study’ at the initial information and consent page of the questionnaire administration.

Data collection

We electronically administered the EDQ followed by three criterion measures (World Health Organization Disability Assessment Schedule (WHO-DAS 2.0) [24, 25], Patient Health Questionnaire (8-item) (PHQ-8) [26] and MOS- Social Support Scale) [27] and a demographic questionnaire using the web-based software Qualtrics [28]. Participants completed the questionnaires in-person via a tablet at the clinical site or remotely via a link in an email or Short Message Service (SMS) text. We administered the EDQ only, again 1 week later. At this time, we asked whether participants had a major change in their health status since their last EDQ completion and if yes, to describe the change in their health.

Questionnaires

Episodic Disability Questionnaire: The EDQ is a newly developed patient-reported outcome measure (PROM) refined from the SF-HDQ [17], comprised of 35-items spanning the six domains: i) physical (10 items); ii) cognitive (3 items); iii) mental-emotional health challenges (5 items), iv) uncertainty or worry about the future (5 items), v) difficulties carrying out day-to-day activities (5 items), and vi) challenges to social inclusion (7 items). For each item, individuals are asked to indicate to what extent they are living with a given health-related challenge on that day (severity scale of 0 to 4), and whether that challenge fluctuated in the past week (episodic score yes or no). The presence score is derived by dichotomizing severity as present (severity 1–4) or absent (severity of 0). See Additional file 1 for the Episodic Disability Questionnaire (EDQ) items.

World Health Organization Disability Assessment Schedule: The WHODAS 2.0 is a 36-item self-administered generic questionnaire of functioning and disability applicable across cultures in adult populations, and directly linked to the International Classification of Functioning, Disability, and Health (ICF) [24, 25]. The WHODAS 2.0 assesses difficulty in performing specific functions over the previous 30 days across six disability domains: i) cognition, ii) mobility, iii) self-care, iv) getting along, v) life activities and vi) participation). Individuals provide an answer for each question on a 5-point Likert scale (range 0–4) with higher scores indicating increasing difficulty completing the task [29]. The WHODAS possesses internal consistency and test–retest reliability and validity and cross-cultural applicability spanning 19 countries [29, 30]. The WHODAS is validated in patients with chronic diseases [31] and people living with HIV [32].

Patient Health Questionnaire: The PHQ-8 is an 8-item measure of depression severity. Items are rated using a Likert-type scale from 0–3, with a total score range of 0–24. A score of 10 or greater is considered major depression, 20 or more is severe major depression [26]. The PHQ-8 is reliable and valid for use with people living with HIV [33].

Social Support Survey Questionnaire: The MOS-SSS is a self-administered 20-item questionnaire designed to measure five different dimensions of social support among patients with chronic illness (emotional/informational support, tangible support, positive social interaction and affectionate support) using 5 response options ranging “none of the time” to “all of the time.” [27] Higher scores indicate higher levels of social support. The MOS-SSS possesses construct validity and reliability with people living with HIV [34].

Demographic Questionnaire: The demographic questionnaire included 26 items comprised of demographic (e.g. age, sex, gender, race), HIV (e.g. date of HIV diagnosis, viral load), and health characteristics (e.g. concurrent health conditions, general health status).

Analysis

We calculated median (interquartile ranges (IQR)) EDQ scores. Severity and presence domain scores were calculated using the algorithm developed through Rasch analysis (score range: 0–100) [17]. Episodic scores included a simple sum transformed on a scale of 0–100. Higher scores indicated greater presence, severity and episodic nature of disability. We calculated median WHODAS 2.0 domain scores, PHQ-8 scores and MOS-SSS domain scores as per guidelines. For the demographic questionnaire we calculated descriptive statistics including frequencies (%) for categorical variables and median and interquartile ranges (IQR) for continuous variables.

Internal Consistency Reliability: We calculated Cronbach’s alpha for time 1 (T1) EDQ domain scores of the severity scale and presence scales with 95% confidence intervals (CIs) (> 0.7 acceptable) [35].

Test–Retest Reliability: We calculated Intra Class Correlations (ICCs) with 95% CIs using T1 and time 2 (T2) EDQ scores estimated from Shrout and Fleiss’ ICC (2,1) (lower bound CI of > 0.7 acceptable) [35]. We calculated ICCs with the entire sample of participants who indicated that they did not have a change in their health status. We then estimated ICCs based on mode of administration either remote (independently via SMS or email link) or in-person (tablet). Our test–retest assessment focused on presence and severity scales of the EDQ as the episodic scale refers to fluctuations in disability in the past week, hence we did not expect consistency in this scale.

Minimum Detectable Change (MDC): We estimated MDC for EDQ domain scores with 90% and 95% confidence as follows [36]: $MDC={z}_{1-\alpha /2}{\widehat{\sigma }}_{baseline}\sqrt{2(1-\rho )}$ where: $\rho$ is the test–retest reliability; $1-\alpha /2$ is the level of confidence; and ${\widehat{\sigma }}_{baseline}$ is the standard deviation of the measure at baseline.

Construct Validity: We examined correlations for 36 primary a priori hypotheses theorizing relationships between EDQ and the WHODAS 2.0 criterion measure subscales, and EDQ scores and known groups of participants completing the EDQ on a good versus bad day. We examined an additional 44 exploratory a priori hypotheses theorizing relationships between EDQ and the PHQ-8 and MOS-SSS criterion measure subscales, self-rated general health status, and known groups of participants living with ≥ 2 versus ≤ 1 concurrent health conditions in addition to HIV (80 hypotheses total). Our construct validity assessment focused on presence and severity scales of the EDQ as the criterion measures do not capture the episodic nature of disability. We derived the a priori hypotheses from earlier construct validity assessments of the HDQ and SF-HDQ [12,13,14]. Spearman correlation coefficients of |≥ 0.30|, |≥ 0.50| and |≥ 0.70|, were defined as ‘weak’, ‘moderate’ and ‘strong,’ respectively [37]. We interpreted the lower and upper bound of confidence intervals when assessing the hypotheses. Construct validity was defined as > 75% confirmed hypotheses [35, 38].

Analysis was conducted using R [@R-psych] [39] and SPSS Software [40].

Sample Size: To detect a weak correlation |r = 0.20|, between EDQ and criterion scores, with a power of 0.90 and alpha of 0.05 required 259 participants [41]. To account for questionnaires with missing responses and loss to follow-up at T2, our targeted sample size was 75 adults living with HIV in each of the five cities for a total of 375 participants [42].

Results

Three hundred fifty-nine participants completed the questionnaires at T1, of which 321 (89%) completed the EDQ at T2. Most participants (80%) completed the T2 EDQ within two weeks of T1 completion.

Of the 321 participants who completed T2, 46 (15%) 274 participants (85%) reported no change in health status and were included in test–retest reliability assessment. The characteristics of those who did and did not complete the EDQ at T2 were similar (see Additional file 2).

Characteristics of participants

See Table 1 of characteristics of participants by site. Most participants identified as men (83%), median age 51 years, living with a median of 4 concurrent health conditions in addition to HIV. There were differences in participant characteristics (Table 1) and EDQ scores across cities, reflecting the types of populations and services provided across the sites. Participants at Casey House, which is a day health program in Toronto tended to be living with more concurrent health conditions, were less likely to be employed, and more likely be on income support compared to participants from the other sites (Table 1). See Additional file 3 for a complete overview of participant characteristic differences across cities. See Additional file 4 for differences in EDQ scores and criterion measure scores across cities.

Table 1 Characteristics of participants by city (n = 359)

Mode of administration

The mode of administration at the Ireland, UK, and US sites were primarily remote whereby participants completed the questionnaires independently online by accessing the link to the questionnaire via SMS text or email (76%), whereas at the Canadian site, most participants completed the questionnaires in-person using a tablet (24%). There were differences in characteristics based on mode of administration at T1 across cities, given the mode of administration was dependent to each city (see Additional file 5).

Internal consistency reliability

The EDQ met criteria for internal consistency across domains of the presence, severity and episodic scales (ICC > 0.7). Cronbach’s alpha for EDQ severity scores ranged from 0.84 (social domain) to 0.91 (day domain), for EDQ presence scores ranged from 0.72 (uncertainty domain) to 0.88 (day domain), and for EDQ episodic scores ranged from 0.87 (physical, cognitive, mental-emotional domains) to 0.89 (uncertainty domain) (Table 2). Lower bound CIs for all Cronbach’s alpha were > 0.70, with the exception of the uncertainty domain (0.68) of the EDQ presence scale (Table 2).

Table 2 Internal consistency reliability for Episodic Disability Questionnaire (EDQ) Domain scores (n = 359 participants)

Test–retest reliability

Overall, the EDQ met criteria for test–retest reliability for EDQ severity domains with ICCs ranging from 0.79 (physical domain) to 0.88 (day domain) and for EDQ presence domains ranging from 0.71 (uncertainty domain) to 0.85 (day domain) (Table 3). Lower bound CIs for all ICCs were > 0.70 with the exception of the physical (0.69) and uncertainty (0.64) domains of the EDQ presence scale (Table 3).

Table 3 Test–retest reliability for Episodic Disability Questionnaire (EDQ) Domains of severity and presence scales (n = 274)

For remote independent administration (n = 209), the EDQ severity domains, met criteria for test–retest reliability with ICCs ranging from 0.79 (physical, uncertainty domains) to 0.89 (day domain) and for EDQ presence domains ranging from 0.72 (uncertainty) to 0.85 (day domain) (see Additional file 4). For in-person administration (n = 24), EDQ severity domains met criteria for test–retest reliability with ICCs ranging from 0.73 (cognitive domain) to 0.87 (social domain) and for two of six EDQ presence domains ranging from 0.48 (uncertainty) to 0.82 (social domain) (see Additional file 6).

Minimum Detectable Change (MDC)

As shown in Table 4, the severity scale for each domain demonstrated highest precision (MDC95% range: 19–25), followed by the presence (MDC95% range: 37–54) and episodic scales (MDC95% range: 44–76) (Table 4).

Table 4 Minimum Detectable Change (MDC) for Episodic Disability Questionnaire (EDQ) Scales (n = 274 participants)

Construct validity hypotheses

Of the 36 primary hypotheses, 29 (81%) were confirmed; and of the 44 secondary exploratory hypotheses, 36 (82%) were confirmed, supporting construct validity for use with adults living with HIV (see Additional file 7).

Discussion

The EDQ scales possess internal consistency reliability, and the EDQ severity and presence scales possess construct validity and test–retest reliability with limited precision and when administered electronically among adults living with HIV across eight clinical settings in four countries, Canada, Ireland, United Kingdom and the United States. This work goes beyond past HDQ measurement property assessment that focused on severity scales only [12,13,14,15,16] and SF-HDQ property assessment with the Rasch interval level of measurement that demonstrated structural validity, reliability and sensibility for use among adults living with HIV [17, 18].

The EDQ presence, severity and episodic scales possessed internal consistency reliability with all lower bound CIs of Cronbach’s alphas > 0.70, with the exception of the uncertainty domain in the EDQ presence scale (0.68) suggesting the domains are homogeneous and collectively measure the broader construct of disability. This aligns with earlier internal consistency reliability assessment of severity scales of the SF-HDQ, whereby Cronbach’s alphas ranged from 0.78–0.85 [17] and original HDQ assessment in Canada and Ireland (Cronbach alpha range: 0.81–0.95) [12]. We anticipated lower Cronbach’s alpha in this study, given the EDQ has fewer number of items in each domain (3–10 items) compared with the original long-form HDQ (3–20 items) [12].

Our assessment for test–retest reliability of the EDQ severity and presence scales demonstrated ICCs > 0.70. Lower bound CIs for all ICCs were > 0.70, with the exception of the physical (0.69) and uncertainty (0.64) domains of the EDQ presence scale (Table 3). While community members living with HIV and clinicians highlighted the utility of the precursor SF-HDQ as an individual assessment of disability [18], this requires higher thresholds for reliability (> 0.80) [43], which we did not achieve in this study. Our assessment of ICCs involving only two points in time, may not account for the potential daily fluctuations in disability, which would have influenced the EDQ scores at T2 interpreted as error, hence the ICCs represented may underestimate the test–retest reliability of the EDQ. Future reliability of the EDQ with more repeated measures over time would be beneficial so that the tool may have utility for administration at routine HIV clinical care visits. Results suggest the EDQ may be positioned for group-based or program evaluation purposes rather than assessing disability or its use as an evaluative measure to assess change at an individual level. Furthermore, we acknowledge the difficulty to identify source of measurement error when the mode of EDQ administration was dependent on the recruitment site with a diversity in populations. Ultimately, clinicians must determine what level of error they are willing to tolerate given the EDQ’s intended use and the types of decisions that might be made based on the scores (e.g. referral to services, eligibility for disability income support, etc.) in clinical practice.

For test–retest reliability, it is important to consider potential sources of error such as mode of administration (electronic or paper-based), type of administration (interview administered versus self-completion), or setting [44]. The primary mode of administration in this study was remote whereby participants received a link via SMS or email to independently complete the questionnaire opposed to in-person administration via tablet at the Toronto site. Lower ICCs reported with in-person (tablet) administration may be attributed to the small sample size (n = 24) (see Additional file 6). Nevertheless, our provision of multiple options for mode of administration is a strength of the study and highlights the utility of the EDQ across different clinical settings.

The EDQ demonstrated lack of precision across the severity, presence and episodic scales (Table 4). This highlights the limitation for the EDQ to measure change in disability as distinct from day-to-day variability. While further work is needed to determine responsiveness to change in disability over time, results suggest the EDQ may only be able to detect large changes in disability that surpass day to day variability and measurement error.

Implications for practice and research

The EDQ has potential for use in clinical and community-based settings to describe disability, facilitate communication among providers and patients, facilitate goal setting, and inform allocation of resources for service provision [6, 8, 45,46,47]. Clinicians and persons living with HIV can use the EDQ to view and interpret the distribution of scores across the domains to specifically indicate what dimensions may pose more (or less) of a challenge.

Our test–retest reliability and construct validity assessment was with the EDQ presence and severity scales only, as we did not expect consistency in a scale that refers to fluctuations in the past week, and the criterion measures did not measure the episodic nature of health challenges. While the episodic scale of the EDQ is an important feature to characterize disability experiences, [16, 18], and is unique to other disability measures [24, 48], the use of the scale should be descriptive in nature to assist in providing a broader picture of the disability experience at a point in time.

This work directly builds on the SF-HDQ and its utility and sensibility with adults living with HIV in community and clinical settings [18]. Clinicians and persons living with HIV may use the EDQ or its precursor SF-HDQ in clinical practice. Both possess the same number of items, structural validity, and scoring algorithm based on the Rasch logit scale [17]. However, we anticipate the EDQ will be of greater use among adults aging with HIV who may experience disability attributed to other concurrent health conditions and not specifically HIV. Furthermore, using the EDQ will enable cross comparisons with other chronic conditions in the future. Current research is assessing the utility and properties of the EDQ with other chronic and episodic conditions [49].

Limitations

We were unable to determine whether mode of administration or site influenced the EDQ properties as we were unable to disassociate mode of administration from clinical site, country, and participant characteristics. Nevertheless, differences in participant characteristics and EDQ scores, reflected the types of populations served at each site and we consider the heterogeneity of the sample across the sites a strength of our study. Only 11% of the sample were women, which under-represents the proportion of women living with HIV in these countries [50, 51]. Research is currently assessing the measurement properties of the EDQ specifically with women living with HIV in the UK [52]. Finally, our results may not be transferable to adults living with HIV in low or middle-income countries [53].

Conclusions

The 35-item EDQ is a newly established generic disability patient-reported outcome measure that measures the presence, severity and episodic nature of disability across six domains (physical, cognitive, mental-emotional, daily activities, uncertainty, and challenges to social inclusion). The EDQ scale possess internal consistency reliability, and the EDQ severity and presence scales possess construct validity and test–retest reliability with limited precision when administered electronically with adults living with HIV across eight clinical settings in four countries. Future work may explore the use of the EDQ in other chronic and episodic conditions.