Archived Comments for:
Evaluation of non-inferiority of intradermal versus adjuvanted seasonal influenza vaccine using two serological techniques: a randomised comparative study
Concerns regarding conclusion of study on evaluation of non-inferiority of intradermal versus adjuvanted seasonal influenza vaccine.
Valter Torri, Mario Negri Institute for Pharmacological Research
5 August 2010
In this study on an elderly population, the immunogenicity of an intradermal seasonal influenza vaccine was compared with that of an adjuvanted vaccine. The primary hypothesis to be tested was that the immunogenicity of the intradermal vaccine would be non-inferior to that of the adjuvanted vaccine for each virus strain in terms of antibody titres determined using HI. Non-inferiority was defined as the upper bound of the 95% confidence intervals (CIs) around the post-vaccination ratios (adjuvanted vaccine/intradermal vaccine) of geometric mean titres (GMTs) being <1.5 for all three strains in the per-protocol set. Based on the stated hypothesis, non-inferiority can only be demonstrated if the null-hypothesis is rejected for all three strains. If this study had been designed to demonstrate non-inferiority for individual strains, significance tests would have had to consider multiplicity adjustments by use of stricter confidence intervals. The primary analysis model in the study did not include these adjustments. Non-inferiority of the intradermal vaccine was claimed for the A/H1N1 and B strains, but not for the A/H3N2 strain using the HI method. It should be noted that the ICH guideline E9 on Statistical Principles for Clinical Trials specifically states that in case of multiple primary variables (e.g., 3 strains) “The primary hypothesis or hypotheses and parameters of interest (...) should be clearly stated with respect to the primary variables identified, and the approach to statistical inference described. The effect on the type I error should be explained because of the potential for multiplicity problems (see Section 5.6); the method of controlling type I error should be given in the protocol.” – yet this is precisely what the authors appear to have not done. The authors also stated that non-inferiority was demonstrated for all three strains in the secondary analysis using by the SRH method, and in a post-hoc analysis of covariance to adjust for difference in baseline antibody titres. This cannot be substantiated, in particular after the primary null hypothesis based on HI test could not be rejected. Furthermore, there was evidence of greater local side effects to the intradermal vaccine with double the risk of at least one solicited injection-site reaction to the intradermal vaccine (70.1%) than the the adjuvanted vaccine (33.8%). In addition, erythema (63.1% vs. 13.4%), swelling (34.2% vs. 8.6%), induration (32.9% vs. 10.6%) and pruritus (28.1% vs 6.5%) were also more frequently reported in the intradermal group. Based on the failure to demonstrate non-inferior immunogenicity together with an apparently higher local reactogenicity, the conclusive statement of the authors that both vaccines are largely comparable seems questionable.
Competing interests
Valter Torri acted as consultant for Sanofi Pasteur and Novartis for review and analysis of articles relative to clinical trials on influenza and cancer vaccines
Author's Reply
Anne Fiquet, sanofi pasteur MSD
11 August 2011
We are grateful to Valter Torri for sharing with us his comments on the conclusive statement of our article and therefore giving us the opportunity to reply.
Before discussing the essence of the comment, we would like to highlight that the results of the study were reported with openness and transparency. This includes for example, making a clear distinction between primary and secondary endpoints and between pre-specified and post-hoc analyses. This is done with the willingness that each reader can fully understand the study results, so did Valter Torri.
The method for controlling type I error rate (Union-Intersection [UI] method) is explained in the protocol which fully complies with the ICH E9 guideline. This is translated in the article by the requirement (repeated several times in the article) that "… the upper bound of the 95% CI around the post-vaccination ratios (adjuvanted vaccine / intradermal vaccine) of GMT being <1.5 for all three strains in the per-protocol set". Technically, the UI method implies an appropriate control of the type II error rate but not multiplicity adjustments of type I error, which remains one sided 2.5% overall.
As stated in the article “Non-inferiority of the intradermal vaccine was not demonstrated for all three strains by the HI method”, however it is of medical interest to study each single non-inferiority hypothesis included in the UI procedure to get a full picture of the results. Sentences stating that non-inferiority of the intradermal vaccine compared with the adjuvanted vaccine was demonstrated for the A/H1N1 and B strains with the HI method do not stand for a formal overall statistical conclusion to the primary objective, but rather correspond to a factual description of the result of each single hypothesis testing, as part of the testing approach.
While considering all reported analyses (primary analysis with a borderline failure, secondary analysis successful based on an endpoint closely related to the primary endpoint, and post-hoc analysis [which only differ from the primary analysis by adjusting on baseline levels] successful on primary endpoint), the authors are fully confident with the conclusion as stated in the article that “both vaccines are largely comparable”.
Regarding injection-site reactions, the authors acknowledge the higher incidence of some local reactions following the use of the intradermal vaccine due to the close proximity of skin surface however there were no differences between the two vaccines for the systemic adverse events reports. Of note most of the local reactions were of mild intensity and transient.
The authors’ conclusion is that the vaccines were “largely comparable” in order to summarize that vaccines are comparable for most endpoints but not all, including safety.
Concerns regarding conclusion of study on evaluation of non-inferiority of intradermal versus adjuvanted seasonal influenza vaccine.
5 August 2010
In this study on an elderly population, the immunogenicity of an intradermal seasonal influenza vaccine was compared with that of an adjuvanted vaccine. The primary hypothesis to be tested was that the immunogenicity of the intradermal vaccine would be non-inferior to that of the adjuvanted vaccine for each virus strain in terms of antibody titres determined using HI. Non-inferiority was defined as the upper bound of the 95% confidence intervals (CIs) around the post-vaccination ratios (adjuvanted vaccine/intradermal vaccine) of geometric mean titres (GMTs) being <1.5 for all three strains in the per-protocol set. Based on the stated hypothesis, non-inferiority can only be demonstrated if the null-hypothesis is rejected for all three strains. If this study had been designed to demonstrate non-inferiority for individual strains, significance tests would have had to consider multiplicity adjustments by use of stricter confidence intervals. The primary analysis model in the study did not include these adjustments.
Non-inferiority of the intradermal vaccine was claimed for the A/H1N1 and B strains, but not for the A/H3N2 strain using the HI method. It should be noted that the ICH guideline E9 on Statistical Principles for Clinical Trials specifically states that in case of multiple primary variables (e.g., 3 strains) “The primary hypothesis or hypotheses and parameters of interest (...) should be clearly stated with respect to the primary variables identified, and the approach to statistical inference described. The effect on the type I error should be explained because of the potential for multiplicity problems (see Section 5.6); the method of controlling type I error should be given in the protocol.” – yet this is precisely what the authors appear to have not done. The authors also stated that non-inferiority was demonstrated for all three strains in the secondary analysis using by the SRH method, and in a post-hoc analysis of covariance to adjust for difference in baseline antibody titres. This cannot be substantiated, in particular after the primary null hypothesis based on HI test could not be rejected.
Furthermore, there was evidence of greater local side effects to the intradermal vaccine with double the risk of at least one solicited injection-site reaction to the intradermal vaccine (70.1%) than the the adjuvanted vaccine (33.8%). In addition, erythema (63.1% vs. 13.4%), swelling (34.2% vs. 8.6%), induration (32.9% vs. 10.6%) and pruritus (28.1% vs 6.5%) were also more frequently reported in the intradermal group.
Based on the failure to demonstrate non-inferior immunogenicity together with an apparently higher local reactogenicity, the conclusive statement of the authors that both vaccines are largely comparable seems questionable.
Competing interests
Valter Torri acted as consultant for Sanofi Pasteur and Novartis for review and analysis of articles relative to clinical trials on influenza and cancer vaccines
Author's Reply
11 August 2011
We are grateful to Valter Torri for sharing with us his comments on the conclusive statement of our article and therefore giving us the opportunity to reply.
Before discussing the essence of the comment, we would like to highlight that the results of the study were reported with openness and transparency. This includes for example, making a clear distinction between primary and secondary endpoints and between pre-specified and post-hoc analyses. This is done with the willingness that each reader can fully understand the study results, so did Valter Torri.
The method for controlling type I error rate (Union-Intersection [UI] method) is explained in the protocol which fully complies with the ICH E9 guideline. This is translated in the article by the requirement (repeated several times in the article) that "… the upper bound of the 95% CI around the post-vaccination ratios (adjuvanted vaccine / intradermal vaccine) of GMT being <1.5 for all three strains in the per-protocol set". Technically, the UI method implies an appropriate control of the type II error rate but not multiplicity adjustments of type I error, which remains one sided 2.5% overall.
As stated in the article “Non-inferiority of the intradermal vaccine was not demonstrated for all three strains by the HI method”, however it is of medical interest to study each single non-inferiority hypothesis included in the UI procedure to get a full picture of the results. Sentences stating that non-inferiority of the intradermal vaccine compared with the adjuvanted vaccine was demonstrated for the A/H1N1 and B strains with the HI method do not stand for a formal overall statistical conclusion to the primary objective, but rather correspond to a factual description of the result of each single hypothesis testing, as part of the testing approach.
While considering all reported analyses (primary analysis with a borderline failure, secondary analysis successful based on an endpoint closely related to the primary endpoint, and post-hoc analysis [which only differ from the primary analysis by adjusting on baseline levels] successful on primary endpoint), the authors are fully confident with the conclusion as stated in the article that “both vaccines are largely comparable”.
Regarding injection-site reactions, the authors acknowledge the higher incidence of some local reactions following the use of the intradermal vaccine due to the close proximity of skin surface however there were no differences between the two vaccines for the systemic adverse events reports. Of note most of the local reactions were of mild intensity and transient.
The authors’ conclusion is that the vaccines were “largely comparable” in order to summarize that vaccines are comparable for most endpoints but not all, including safety.
Competing interests
I am an employee of sanofi pasteur MSD.