Development of the Flu-PRO: a patient-reported outcome (PRO) instrument to evaluate symptoms of influenza

Study design and setting

A two-stage qualitative instrument development methodology was followed, involving adults (≥18 years) with laboratory-confirmed influenza recruited through clinical sites in the US and Mexico. Stage I involved concept elicitation interviews with the target population to gather information regarding patient experience of influenza symptoms, including type, magnitude, expression, pattern of onset, and recovery. From this, a draft item pool and conceptual framework were developed for the new Flu-PRO instrument, including its content and structure (item phrasing, length, response options, recall, instructions). Stage II consisted of cognitive interviews to assess completeness, comprehension, and interpretability of the draft Flu-PRO instrument from the respondent’s perspective. Qualitative, semi-structured telephone interviews were utilized in both stages. Patients with laboratory-confirmed influenza were recruited from military treatment facilities or associated clinics in the US (six clinics in San Diego, CA; Portsmouth, VA; Tacoma, WA; San Antonio, TX; Jacksonville, NC; and Bethesda, MD), and the Mexico Emerging Infectious Diseases Clinical Research Network (LaRed) in Mexico, which consisted of three hospitals: Instituto Nacional de Ciencias Médicas y Nutrición Salvador Zubirán (Nutricion), Instituto Nacional de Enfermedades Respiratorias (INER), and Hospital General Dr. Manuel Gea. The protocol for both stages was approved by Institutional Review Boards at the Uniformed Services University and at both the US and Mexico sites.

Stage I: concept elicitation interviews

The purpose of the concept elicitation interviews was to elicit participants’ descriptions of their influenza symptoms to inform the content and structure of the new measure. The following inclusion criteria were used to recruit potential participants from US sites: 1) aged 18 years or older; 2) eligible for care in Department of Defense facilities (DEERS eligible); 3) symptoms consistent with influenza-like illness (ILI): a) Fever with body temperature ≥100.4 °F [38 °C] AND respiratory symptoms OR b) Subjective fever AND respiratory Symptoms; 4) able to speak in English; and 5) able to provide written informed consent. Respiratory symptoms were defined as: cough, or sputum production, or shortness of breath, or chest pain, and/or sore throat.

Exclusion criteria were: 1) self-report of concurrent asthma or chronic obstructive pulmonary disease (COPD) exacerbation; 2) concurrent pneumonia symptoms with clinical evidence confirming bacterial pneumonia, i.e., pleural effusion or empyema, consolidated single lobar pneumonia documented by radiological testing with otherwise normal lung fields, or bacteremia with a pathogen known to cause pneumonia; 3) any other relevant condition that would interfere with participants’ ability to participate in a one-on-one telephone interview; and 4) complicated pregnancy.

For the Mexico participants, the following inclusion criteria were used to recruit participants: 1) aged 18 years or older; 2) enrolled in an existing influenza natural history study at the Mexico sites that included swab for influenza testing; 3) Spanish speaking; and 4) able to provide written informed consent. Exclusion criteria were: 1) any relevant condition that would interfere with their ability to participate in a one-to-one telephone interview; and 2) declined consent to having their interview audio-record interview.

All participants with ILI who met these criteria were enrolled and interviewed for this study; however, only those who were later shown to have a laboratory-confirmed influenza diagnosis were included in the analysis. This ensured the concepts elicited in developing the new questionnaire were from participants known to have confirmed influenza, rather than another ILI. Both inpatients and outpatients were eligible for inclusion. In both the US and Mexico, all participants’ influenza diagnostic procedure was performed as part of existing studies’ protocols and not specifically for the purpose of the Flu-PRO development study. These tests included rapid antigen testing (RAT) and real time reverse-transcription polymerase chain reaction (RT-qPCR) for influenza. Available results classified patients as influenza positive or negative, but did not further classify patients by type of influenza, as the medical literature poses the symptoms of influenza are similar across types of influenza and types of ILI.

Participants provided written informed consent at the recruitment site and were scheduled for a telephone interview with the study research team. A semi-structured interview guide was used to facilitate the interviews, with open-ended questions asking participants to describe their symptoms and characterize their nature, severity, onset, duration, variability, and indictors of recovery. Information on the importance or impact of symptoms was gathered to understand the relative importance of symptoms from a content validity perspective.

All interviews were conducted by trained researchers. US interviews were conducted in English by research staff from Evidera (Bethesda, MD) and interviews in Mexico were conducted in Spanish by staff from the LaRed network using the translated interview guide. At the end of the interview, participants were asked sociodemographic and clinical/health-related questions, with data used to characterize the sample. Interviews were transcribed verbatim, with the data cleaned and de-identified for analyses. Mexico interviews were translated by an external translation company (FACIT - Functional Assessment of Chronic Illness Therapy group; Chicago, IL) for analyses in English.

A coding dictionary was developed by examining initial transcript data to identify key themes and constructs participants expressed during the interviews. The software program ATLAS.ti (version 5.0) was used to facilitate the systematic analysis of data. All transcripts were then coded thematically and reviewed by a second analyst, with any discrepancies resolved through discussion with the primary or senior analyst. Symptoms that emerged from the interviews were tracked on a grid to monitor saturation, defined as the point at which additional sampling offered no new information [9]. US data from adult participants were collected and analyzed first and to the point of saturation. Data collected from Mexico were analyzed using the same coding dictionary, assessing symptoms reported and noting any differences with US results.

Development of the Flu-PRO instrument

The multi-disciplinary Flu-PRO development team used results of the concept elicitation interviews to draft an English version of the Flu-PRO instrument, including items, instructions, recall period, and response options. This ensured the instrument was grounded in the words and phrases that patients with influenza commonly use to describe the symptoms they experienced. A translatability assessment was performed by an expert in PRO instrument translation and ePRO formatting to ensure translatability of words and phrases, and suitability of conversion to ePRO format.

An item definition dictionary and translation instructions were then developed and the draft Flu-PRO instrument was translated into Spanish using methods to ensure conceptual equivalence [10]. The steps involved in using this method were: 1) two forward translations by native speakers of Spanish; 2) one reconciled version of the two forwards by a third native speaker of Spanish; 3) one back translation of the reconciled version into English by an English speaker fluent in Spanish; 4) comparison of source text and back-translated text to identify possible discrepancies by FACIT staff; 5) review of the translations and FACIT comments by the bilingual language coordinator (LC) for Spanish; 6) finalization of the translations by the LC for Spanish; 7) quality review and resolution of any outstanding harmonization issues by FACIT; and 8) formatting by FACIT staff and proofreading of the final questionnaire by two professional translators who are native speakers of Spanish. Prior to its use in the cognitive interviews, investigators from Mexico reviewed the translated version to assure consistency with the words and phrases participants from Mexico used during the elicitation phase.

Stage II: cognitive interviews

The purpose of the cognitive interviews was to assess subject comprehension and interpretability of the Flu-PRO, its ease of use, and completeness. The exclusion criteria and recruitment processes were the same as those used in the concept elicitation interviews outlined above. However, only participants with laboratory-confirmed influenza were enrolled within two weeks of diagnosis. A semi-structured interview guide was used to facilitate the cognitive interviews; questions assessed content coverage and participants’ understanding and interpretation of the Flu-PRO questionnaire (instructions, items, response options, recall period) with an open-ended question asking if there were any additional symptoms the person had experienced that were not covered in the questionnaire. As before, interviews were audio-recorded and transcribed verbatim. Sociodemographic and clinical/health-related questions were asked at the end of the interview with data used to characterize the sample. Mexico interviews were conducted in Spanish, transcribed, and translated into English for analyses. Item-by-item analyses were performed to assess clarity and consistency in interpretation across subjects, and to determine the need for adjustments or editorial changes before the instrument was finalized for quantitative evaluation.

Stage I: concept elicitation interviews

Flu-PRO development

Symptoms were reviewed and discussed by the multi-disciplinary Flu-PRO team. Candidate items, response options, and structure (e.g., instructions and format) were developed, together with a conceptual framework for the instrument, using body systems as the organizational structure (see Fig. 1). For ease of use, a 5-point severity scale was selected, from “not at all” to “very much” for 32 items; 5-point frequency scales were used for vomiting and diarrhea (0 times to 4 or more times) and sneezing, coughing, and productive cough (never to always). Symptoms that were conceptually similar but worded differently by different participants were included in the item pool in order to subject the alternatives to qualitative and quantitative testing. For example, “scratchy or itchy throat”, “sore or painful throat”, “swollen throat”, and “difficulty swallowing” were all used to describe symptoms associated with a sore or inflamed throat. Rather than select one, all were included as candidate items for empirical testing. The instrument was formatted for ease of completion by paper-pen in a clinic or home setting, with minimal wording, check boxes, and gridlines. The format was also intended for ease of conversion to electronic format, including web-based or tablet, with no change in wording and minimal adjustments in appearance [11].

Stage II: cognitive interviews

Understanding, saturation, and endorsement

Participants in both the US and Mexico samples had a good understanding of the instructions, recall period, response options, and the majority of the symptom items on the questionnaire. Participants considered the content and items to be relevant to their recent and/or previous influenza experiences. No new symptoms arose during the interviews, indicating symptom saturation had been reached during Stage 1.

As expected, participants identified symptom items they considered overlapping and/or highly related: “sore throat”, “swollen throat”, and “difficulty swallowing” (n = 5 US; n = 4 Mexico); ‘felt lightheaded” and “felt dizzy” (n = 5 US; n = 2 Mexico); and “head congestion”, “sinus pressure”, and “headache” (n = 4 US; n = 11 Mexico). Additionally, two participants from the US identified “sore and painful eyes” and “eyes sensitive to light” as overlapping. Participants from Mexico also identified the following overlapping/highly related symptoms: “sinus pressure”, “nasal congestion”, and “runny or drippy nose” (n = 10); “trouble breathing” and “nasal congestion” (n = 6); “chest tightness”, “pain with breathing or coughing”, and “muscular/general pain” (n = 5); and “body aches or pains” with “other” symptoms (n = 4). Because these comments were expected and raised by a relatively small number of people in this qualitative study, all of the items were retained for testing in a large, quantitative study, and flagged for additional consideration.

Stage III psychometric testing is currently underway and as part of this process, patients were timed in filling out the questionnaire. The total time to fill out the questions was approximately five minutes, showing low respondent burden and ease of use for clinical research studies.

Figure 2 shows the most frequently reported influenza symptoms by country, organized by body system. The symptoms participants rated as most severe, by country, are shown in Fig. 3.