Study design and participant recruitment

A survey with two questionnaires, one per time point, was carried out in order to evaluate usage, user satisfaction and the effect of the mobile ‘Tekenbeet’ app. According to Dutch law, this general Internet-based survey involving healthy volunteers from the general population requires no formal medical ethical approval.

During the first data collection period (Additional file 1: Questionnaire 1), respondents were recruited through several channels. To fill out the questionnaire, it was not necessary for respondents to have downloaded the app or to even be aware of its availability; anyone willing to participate could do so. A call for participants for this study was posted on the RIVM website (https://www.rivm.nl/tekenbeet), which attracts around 220,000 unique visitors annually, who are most likely interested in information on ticks and LB. Therefore this group of visitors represents a large part of the target group on which this mobile app is focused, and from which it is important to get relevant results. In addition, several other external websites from our stakeholders such as national nature organizations, Scouting Netherlands and the Foundation Tick Bite Diseases advertised this call for participants. Furthermore, several organizations included a call in their subscriber newsletters. After filling out questionnaire 1, respondents who were willing to fill out a second questionnaire (Additional file 1: Questionnaire 2) a few weeks later, could leave their e-mail address. To increase response rates, participants could win one of 50 gift certificates with a value of 10 Euro if they filled out both questionnaires.

The first questionnaire (Additional file 1: Questionnaire 1) could be filled out from 18th July till 1st October 2014; the second questionnaire (Additional file 1: Questionnaire 2) from 6th till 30th October 2014. The first questionnaire had a much longer inclusion time compared to the second, which was due to the inclusion methods. During the first questionnaire, respondents were passively recruited. During the second questionnaire, respondents who indicated that they were willing to participate left their e-mail address at the end of the first questionnaire and could therefore be contacted directly. It was important to collect the data before the end of the tick season (end of October) to ensure the content of the questionnaire was still relevant. It was not the goal to randomly allocate the respondents over the two groups (i.e. those who did and those who did not download the app), since it was impossible to force respondents to refrain from downloading the app as it is publicly available. Therefore respondents were assigned to a group based on their willingness to download the app or not. After agreeing to fill out the questionnaire, respondents were asked if they were willing to download the app if they did not already have the app. Respondents who already had the app or were voluntarily willing to download the app were included in the app user group (the group of people who already downloaded the app or did so in the beginning of the first questionnaire). Respondents who were not willing to download the app or did not have a smartphone were included in the non-app user group (the group of people who did not download the app).

Questionnaires

The questionnaires (Additional file 1) both consisted of two parts; the first part intended to evaluate usage and user satisfaction of the app, which was only intended for those in the app user (‘downloaders’) group. The second part was filled out by respondents in both the app user and the non-app user group, and intended to evaluate the effects of having the app, or not, on determinants such as knowledge, perceived severity and susceptibility, self-efficacy, response efficacy and intention to comply with measures to prevent tick bites and LB. The different utilities of the app correspond with these determinants. During the first questionnaire, respondents in the app user group who had only just downloaded the app were asked to take some time to explore the app. Respondents in the app user group who already downloaded the app before participating in this survey were assumed to have familiarized themselves with the app’s content.

To evaluate usage and user satisfaction, app user group respondents were asked to give feedback on the app (Additional file 1: Questionnaire 1). This involved questions about usefulness of different elements of the app, how many times the app was used (if downloaded prior to participation in the survey), and whether one would recommend the app to others. In the second questionnaire (Additional file 1: Questionnaire 2), questions about usage and user satisfaction were asked again to assess the captivating effect of the app. This time the questions were about ongoing usage and user satisfaction since respondents had access to the app for a longer period of time.

For the effectiveness evaluation, the effects of the app on respondents’ knowledge, perceived severity, perceived susceptibility, self-efficacy, response efficacy, and intention regarding the prevention of tick bites and LB were assessed. These determinants were derived from the Protection Motivation Theory (PMT) and the Health Belief Model (HBM) and correspond to our previous evaluation studies of an educational game, leaflet and movie on ticks [15, 16, 20, 21]. Different components of the app are expected to influence the following psychosocial determinants: the information on ticks and LB is expected to influence knowledge and perceived severity, information on checking and removing ticks is expected to influence response efficacy and self-efficacy and intention, and tick radar information is expected to influence perceived susceptibility. The questionnaire was developed further on the basis of an existing questionnaire already used in a study to evaluate the effect of a movie and a leaflet on prevention of tick bites and LB in the Netherlands [22]. In this questionnaire, different psychosocial determinants were based on parts of the PMT and the HBM [20, 21]. The PMT proposes that intention to protect oneself against a health threat is based on the threat appraisal and the coping appraisal, which each depend on two factors. The appraisal of a health threat, which is also part of the HBM, consists of perceived severity (How severe are the consequences of the disease?) and perceived vulnerability (How probable is it that I will contract the disease?). The coping appraisal consists of response efficacy (How effective is the recommended behavior in avoiding the negative consequences?) and self-efficacy (To what extent am I able to perform the recommended behavior successfully?) [23]. To assess whether there was a possible lasting effect of the app on the different determinants, questions to evaluate effectiveness were asked in both the first and the second questionnaire.

In this questionnaire, level of knowledge was measured with eight true/false questions (e.g. ‘Ticks usually fall out of trees to bite.’). The other psychosocial determinants were measured on a 7-point Likert scale, ranging from 1 (totally disagree) to 7 (totally agree). Response efficacy was measured with five questions (e.g. ‘Checking for ticks after every outdoor visit will help prevent LB.’), self-efficacy with five questions (e.g. ‘I am capable of recognizing a tick on my body.’), intention with four questions (e.g. ‘I am planning to check my skin for ticks after every outdoor visit.’), perceived susceptibility with two questions (e.g. ‘How likely do you think it is that you will be diagnosed with LB within the next year?’), and perceived severity with three questions (e.g. ‘If I were to sustain a tick bite, I would be worried about the possible consequences.’). Personal demographics were recorded, including gender, age, and level of education achieved. In addition, questions about everyday use of mobile apps, amount of time spent outdoors (increased risk of tick bites), and having children aged 0–17 years at home (increased tick awareness) were asked.

Statistical analyses

Data from questionnaires 1 and 2 were analyzed using SPSS 22 for Windows. To assess user satisfaction, means (standard deviation (SD)) and proportions were used. The study population was described using means (SD) and proportions. Sum scores were calculated for questions on knowledge. If psychosocial items showed sufficient internal consistency, they were measuring the same construct, and were therefore assembled into a single construct. Since Cronbach’s alpha is an inadequate estimate for both validity and reliability, omega was used to assess the internal structure of the items in each scale [24]. Perceived severity was assessed by means of 3 items (Ω = 0.90, 95% CI = 0.88–0.92), perceived susceptibility by means of 2 items (Ω = 0.56, 95% CI = 0.46–0.64), self-efficacy by means of 5 items (Ω = 0.72, 95% CI = 0.67–0.77), response efficacy by means of 5 items (Ω = 0.80, 95% CI = 0.77–0.84), and intention by means of 4 items (Ω = 0.69, 95% CI = 0.63–0.75). The full text of all items can be found in Appendix 1.

After dichotomization of the items on the 7-point scale, odds ratios (ORs) with 95% confidence intervals (CI) were calculated, with a P value from a Chi-square test, to find associations between psychosocial determinants and the two groups. When univariate analysis showed significant p-values (p < 0.05), multivariate logistic regression analyses, adjusted for gender, having children aged 0–17 years at home, and having experience of tick bites, was used to calculate adjusted ORs. Associations between the different determinants were corrected for multiple testing using the Benjamini-Hochberg method [25].

Response

A total of 812 respondents started filling out the questionnaire, while 258 did not finish the complete questionnaire, resulting in 554 complete questionnaires. Most respondents were recruited during the first few weeks of the inclusion period. Of the 554 respondents, 340 gave details of their e-mail address. On 6th October 2014 they received an invitation e-mail requesting completion of the second questionnaire. A reminder e-mail was sent on 16th October. Seventy percent of the invitees (238 out of 340) completed the second questionnaire (Fig. 4).

Respondents

Sixty-three point four percent (n = 351/554) of the respondents was female and the mean age was 46.6 years (SD: 13.5). Sixty-two point three percent (n = 345/554) had a high educational level (defined as higher education or university), and 33.9% (n = 188/554) had children aged 0–17 living at home. Eighty-seven point five percent (n = 485/554) had tick bite experience themselves or through someone they knew, whereas 56.5% (n = 313/554) had LB experience themselves or trough someone they knew. Twenty point four percent (n = 113/554) had a profession that involved working outdoors. Most of the respondents, 88.7% (n = 490/546), indicated that in the three months prior to filling out the questionnaire they sometimes or always checked for tick bites following potential exposure.

Usage and user satisfaction of the app

Just over 8 % (8.3%; n = 46/554) of the respondents indicated that they did not own an app-enabled mobile phone. Therefore, these respondents were excluded in the questions about app usage and user satisfaction. Forty-five point one percent (n = 229/508) of the respondents who owned a mobile phone indicated they were aware of availability of the ‘Tekenbeet’ app and 27.8% (n = 141/508) had already downloaded the app prior to the survey. An overview of how often these respondents used the app in the recent past (ranging from one to ten weeks) can be found in Fig. 5. Thirty-one point one percent (n = 158/508) were planning to download the app. When these participants were asked to download the app at that moment, 64.6% (n = 102/158) of them did so. In total, 43.9% (n = 243/554) had the ‘Tekenbeet’ app on their mobile phone at the moment of the first questionnaire.

Most participants, 78.2% (n = 190/243), indicated that they thought information in the app was presented as expected. In questionnaire 2, user satisfaction (range from 1 to 10) mean score was 7.35 (SD 0.96). A near identical score to questionnaire 1 was achieved for recommendation of the app to others in questionnaire 2, at 90.6% (n = 87/96). More than half of the respondents in questionnaire 2 (53.1%; n = 60/113) indicated that they used the app 1–3 times since they downloaded it.

Effects of the app (Additional file 1: Questionnaire 1)

Respondents in the app user group scored significantly more often high on knowledge compared to respondents in the non-app user group (OR 3.37; 95% CI 2.15–5.28); the same applied to self-efficacy (OR 2.67; 95% CI 1.29–5.55). The app user group more often had a more positive intention to comply with measures to prevent tick bites and LB when compared to the non-app user group (OR 2.47; 95% CI 1.14–5.36); a similar impact was observed for response efficacy (OR 1.74; 95% CI 1.02–2.99). There was no significant difference between the two groups in perceived susceptibility or severity scores.

Multivariate logistic regression analyses showed that gender, having children aged 0–17 years at home, and having experience with a tick bite, had only a very minimal effect on the significance of the relationship between the determinants and the two groups for both intention and knowledge. However, response efficacy and self-efficacy were no longer significantly different in both groups. The results of the multivariate analyses can be found in Table 4.

Effects of the app (Additional file 1: Questionnaire 2)

Results of questionnaire 1 compared to questionnaire 2

Limitations and future research

We were pleasantly surprised to learn that almost half of the respondents indicated that they were aware of the availability of the ‘Tekenbeet’ app before participating in the survey. An invitation to participate in the survey was posted on websites with a special interest in nature or ticks and LB, which was also the place where the app was promoted in the months prior to the survey. This may explain why so many respondents were aware of the availability of the app. However, being aware of the availability of the app due to promotion does not necessarily mean someone is willing to download the app. This can be explained by many reasons such as not having a smartphone or simply not being interested in having an app on this subject. Furthermore, almost a third of the respondents who started filling out questionnaire 1 did not complete it, most of them quit after only one or two questions. These factors may have introduced selection bias, but due to this kind of inclusion method, a non-response study was not possible. This recruitment method was chosen to increase the likelihood that native tick-interested people participated in the study, and only native app-interested people would fill out the questions about the app, since these people are precisely the target group of the mobile app. Clearly, the recruitment method used in this study does not provide a representative sample of the Dutch population. However, we were mainly interested in the usage and the effects of the app on the different determinants of people who, in real life, would also use this app. To reduce selection bias, in the multivariate analyses we corrected for gender, having children aged 0–17 years at home, and having experience of tick bites. In the study population there was an overrepresentation of women and people with a higher education level. However, this phenomenon is seen more often when respondents are recruited through websites or social media [27].

People had the option to download the app if they were willing to do so. They were not divided into two groups beforehand, and this may have contributed to selection bias. However, in this study, we were mostly interested in respondents that showed an unforced interest in ticks/LB and/or in downloading the app.

At the end of each questionnaire, respondents could leave comments. This resulted in some useful recommendations to improve its user-friendliness. For example, the inclusion of a functionality to make it possible to send tick diary data to someone else if approved by the user, such as parents or one’s general practitioner. This could be an improvement, especially for teachers or supervisors at children’s nature clubs. Furthermore, the tick radar function in the app was considered ‘least useful’ by 20.2% of respondents, mostly due to the unavailability of a zoom function. These feedback results may be acted upon to make improvements to future updated versions of the app. The introduction of a zoom option for the tick radar is currently being considered.

Future research might be helpful to determine the effect of the app even more. As mentioned before, it is a challenge to reach people who are less intrinsically interested in this subject. Research on how to reach a less engaged and motivated population could be beneficial to achieve the apps’ full potential. Furthermore, using a baseline evaluation might clarify even better how much the application improves knowledge compared to using non-app users as a proxy as was the case in this study.

Finally, a follow-up study could be helpful to determine whether the intention to apply tick prevention measures actually leads to increased tick preventing behavior. However, there is only a limited timeframe in which ticks are active, meaning a follow-up study has to take place a year later.

After such a period, most respondents have probably already forgotten about if and when they were outdoors and if they checked for ticks. Therefore, long periods between measurements would probably not measure the influence of the app anymore and result in recall bias. This can be solved by measurements in quicker succession. Even so, participants in the non-app user group would have the opportunity to download the app in the meantime, which may affect the results of the study.