Implications of Oncology Trial Design and Uncertainties in Efficacy-Safety Data on Health Technology Assessments

Round 1

Reviewer 1 Report

The authors should be commended on embarking on a review of a unique and interesting topic pertaining to the conduct of HTA in oncology. However, the discussion within the review was largely focused on issues with oncology data with limited discussion of how this impacts HTA (as opposed to other decisions), as the impact on HTA are expected to be unique. The authors also begin discussion of how HTA use in oncology may differ between different countries and the unique limitations which may be inherent to different countries; however, further expansion on this interesting point is lacking. Finally, it is unclear from reading this manuscript how the literature review was used to shape the discussion in the manuscript. Overall, this manuscript would benefit from restructuring with clearly defined objectives for the review and a more systematic approach to the literature search, and subsequent presentation of the findings. 

Minor comments:

1. Abstract - Methods & Results: please expand on methods (what databases, the goal of the search, what was important inclusion/exclusion criteria for review, years, etc). Without this, the reported results seem disjointed and seem opinion-based (as opposed to 'results'). 

2. Abstract: what is meant by terms such as efficacy-safety or clinically-economically? I suspect this means either or but not clear as is. Consider revising. 

3. Introduction - line 44-48: Two sentences seem to be contradictory. Consider rephrasing. 

4. Table 1: Should be adjusted to improve readability. For instance, more consistent descriptors should be employed (e.g. under Evidence, list randomized control trials and non-randomized trials; this could be summarized as clinical trials (as is done in column 2)). Listed points should also be mutually exclusive (e.g. assessing value for money is done to inform population coverage/reimbursement decisions). Finally, the table may require additional information (e.g. in Outputs, cost-effectiveness, budget impact, etc should also be added to HTA column). 

5. Introduction - line 71: A critical part of the HTA process is to address the evidence quality and describe the uncertainty. This would inform whether early investment/adoption is 'good' or 'bad'. The way the first sentence is worded does not capture this strength of HTA. Suggest re-wording. Also, not sure this sentence fits with the theme of the paragraph it introduces which talks about the impact of low-value care. 

6. Section 1.1: As it reads, it is uncertain what type of review this is (narrative vs. scoping). It is also uncertain what the objective of the review is. Suggest making more explicit and adjusting the terminology used to describe the methods. 

7. Figure 1: how was this generated? The included link does not link to a webpage that includes/describes the graph. 

8. Figure 2: does not link the corresponding discussion in section 2.1 which aims to describe the multi-attributes of value. Consider revising. 

9. It is unclear how section 4 & 5 is distinct from section 3. From how it reads, the listed points in section 4 & 5 seem to pertain to additional challenges with evidence available to conduct HTA. 

10. Section 3/4/5: discuss overall problems with clinical trial data. The discussed points could be made for challenges with clinical trial data to inform regulatory approval, clinical decision-making, as well as HTA. What is missing in these sections is what the specific problem is for HTA processes and perhaps, how HTA can 'overcome' or 'account' for data limitations.

Author Response

Reviewer 1.

The authors should be commended on embarking on a review of a unique and interesting topic pertaining to the conduct of HTA in oncology. However, the discussion within the review was largely focused on issues with oncology data with limited discussion of how this impacts HTA (as opposed to other decisions), as the impact on HTA are expected to be unique. The authors also begin discussion of how HTA use in oncology may differ between different countries and the unique limitations which may be inherent to different countries; however, further expansion on this interesting point is lacking. Finally, it is unclear from reading this manuscript how the literature review was used to shape the discussion in the manuscript. Overall, this manuscript would benefit from restructuring with clearly defined objectives for the review and a more systematic approach to the literature search, and subsequent presentation of the findings. 

We thank the expert reviewer for the careful assessment of the paper and the in-depth comments. We provided actions to the single issues raised in this report.

 

Minor comments:

1. Abstract - Methods & Results: please expand on methods (what databases, the goal of the search, what was important inclusion/exclusion criteria for review, years, etc). Without this, the reported results seem disjointed and seem opinion-based (as opposed to 'results'). 

Thanks for the important point. We expanded on the preparatory material, the datasets researched for the scoping, and the final selection of literature. This is not a systematic review, and the initial scoping research was done by one lead author (DT) for identifying major gaps with a preliminary data extraction, and literature mapping. The mapped terms have been reported in the paper. The timeline of research is for all from database inception to May 15^th 2022, with a focus on the last 5 years. In a narrative review, preparatory materials, preliminary research and timelines are not commonly specified, including as in this Journal, that is why the authors felt they could be skipped for now and did not track the selection process per PRISMA as in systematic reviews.

 

2. Abstract: what is meant by terms such as efficacy-safety or clinically-economically? I suspect this means either or but not clear as is. Consider revising. 

Thanks, that has been revised. We specified what is the term of clinical and economical outcomes, alias the major focuses of HTA discipline.

 

3. Introduction - line 44-48: Two sentences seem to be contradictory. Consider rephrasing. 

Thanks for highlighting. The contradiction in terms has been amended. The non-contrast nature of the two sentences has been now amended, to make sure it is not misunderstood as conflictual.

 

4. Table 1: Should be adjusted to improve readability. For instance, more consistent descriptors should be employed (e.g. under Evidence, list randomized control trials and non-randomized trials; this could be summarized as clinical trials (as is done in column 2)). Listed points should also be mutually exclusive (e.g. assessing value for money is done to inform population coverage/reimbursement decisions). Finally, the table may require additional information (e.g. in Outputs, cost-effectiveness, budget impact, etc should also be added to HTA column). 

This is a very relevant point. All the suggestions have been implemented.  

 

5. Introduction - line 71: A critical part of the HTA process is to address the evidence quality and describe the uncertainty. This would inform whether early investment/adoption is 'good' or 'bad'. The way the first sentence is worded does not capture this strength of HTA. Suggest re-wording. Also, not sure this sentence fits with the theme of the paragraph it introduces which talks about the impact of low-value care. 

We thank the expert reviewer for the in-depth and careful review of this part. We rephrased, according to the inputs received.

 

6. Section 1.1: As it reads, it is uncertain what type of review this is (narrative vs. scoping). It is also uncertain what the objective of the review is. Suggest making more explicit and adjusting the terminology used to describe the methods. 

Thanks, the review is a narrative type. Objectives have been better outlined now.

 

7. Figure 1: how was this generated? The included link does not link to a webpage that includes/describes the graph. 

Thanks, the image was constructed based on WHO data. We now included it as a reference.

 

8. Figure 2: does not link the corresponding discussion in section 2.1 which aims to describe the multi-attributes of value. Consider revising. 

Thanks for noting, we rephrased and reviewed the paragraph and the link to the image.

 

9. It is unclear how section 4 & 5 is distinct from section 3. From how it reads, the listed points in section 4 & 5 seem to pertain to additional challenges with evidence available to conduct HTA. 

Thanks for highlighting this point. We first commented on general issues in conducting HTA, mostly focused on higher level clinical trial design (eg, endpoints, populations enrolled). Then, as titled, we focused on clinical trial designs specifically, with case studies.

 

10. Section 3/4/5: discuss overall problems with clinical trial data. The discussed points could be made for challenges with clinical trial data to inform regulatory approval, clinical decision-making, as well as HTA. What is missing in these sections is what the specific problem is for HTA processes and perhaps, how HTA can 'overcome' or 'account' for data limitations.

This is well noted. We expanded on this part.

Reviewer 2 Report

This is a interesting overview paper on the problems encountered while performing an HTA for cancer drugs. The weakness I see in this paper is, that most of the issues covered have more to do with performing clinical trials and the decision making on drug approval based on these trials, then with performing an HTA as such. HTAs can only be as good as the evidence which inputs the HTA. This evidence is mostly of low or lower quality not only in cancer drugs but in many technologies (for example diagnostic tests, or surgery), increasing the uncertainty in general. For me therefore an overall clear message to the HTA community is somewhat missing.

Further specific remarks:

References:

·         Sometimes there is a space between a word and the following reference (line 63: money [9].) or there is no space (line 58: innovation[7]) Check it through and make it for all references in the one or the other way.

·         The end point of a sentences is marked with a “.” sometimes before the reference (line85: individuals.[13]) or after the reference (line 81: countries [12].) Check it through and make it for all references in the one or the other way.

·         In case of:  .[24],[23][25] use [24-25], or [27][28][29][30][31] use [27-31]

·         [57][55][58][43]. Consider the order 43,55,57,58. Also for other references like [2][65][34]

Table 1: would be nice to have it all on one page?

Line 71-73: This sentence is very difficult to understand. HTA itself (because of missing best information) does not invest in low-value care or yield in poor population health outcomes. HTA may reveal that best information is missing and the technology, when used and reimbursed is of no value.

Line 85-87: wasteful spending itself does not impact cancer outcomes….. it’s the non effective drugs that do so?

Line 92-94: HTA itself cannot readjust the trajectory if low-value interventions. That can only be done by the health policy makers. They need to decide, based on the outcomes of a HTA, if they want to lessen wasteful spending and remove low-value interventions from a health program.

Line 133. Remove the “.” after implication

Figure 2 (and 3) is of a bad graphical quality, try to improve

Line 151: “for example before market» unclear what you mean

Line 184: “direct clinical endpoints.» what do you mean by direct clinical endpoints? Please give some examples like overall survival

Line 195-197. “clinically meaningful improvement” I do agree that “OS” belongs to this, however many oncologist will not agree that PFS is NOT a clinically meaningful improvement! Being progression free can be highly important to patients and their awareness of their health state.

Line 216-217: do not use “Also” twice in the same sentence. And do you mean “whether or not”?

Line 319 “regardless of TMB high» should be “regardless of high TMB” ?

Line 339 and 340: “deliver effective and sustainable HTA decisions” in my understanding HTA does not deliver decisions. It delivers the basis for policy makers to make “a decision”.  I would not write “One contributor to the lack of adequate data to inform HTA decisions” but for example “One contributor to the lack of adequate data to inform HTA outcomes” or HTA results.

Line 365-366: Although this is true, HTA can only be as good as the evidence which is used, so if clinical trials and approval of drugs stay on the same level as it is now, harmonising HTA requirements won’t solve the problem….

Line 412-417: I do not understand what risk-sharing and outcome-based agreements have to do with performing a HTA. These are deals between health insurance, government and pharmaceutical companies and should not influence the production of a HTA. The whole paragraph somehow seems to mix up things, putting HTA in the context of performance-based systems. HTA should be an independent process of evaluating all possible outcomes, completely independent of what agreements on pricing or made.

Line 434-440: Again, I would not use “HTA decision”. It’s not the HTA who makes a decision. Again, use HTA Outcome.

Author Response

Reviewer 2.

This is a interesting overview paper on the problems encountered while performing an HTA for cancer drugs. The weakness I see in this paper is, that most of the issues covered have more to do with performing clinical trials and the decision making on drug approval based on these trials, then with performing an HTA as such. HTAs can only be as good as the evidence which inputs the HTA. This evidence is mostly of low or lower quality not only in cancer drugs but in many technologies (for example diagnostic tests, or surgery), increasing the uncertainty in general. For me therefore an overall clear message to the HTA community is somewhat missing.

Thank you for the expert inputs, we agree that an HTA perspective deserves better articulation. We expanded the discussion sections, to address this point.

Further specific remarks:

References:

• Sometimes there is a space between a word and the following reference (line 63: money [9].) or there is no space (line 58: innovation[7]) Check it through and make it for all references in the one or the other way. Thanks, all spaces have been added where missing.
• The end point of a sentences is marked with a “.” sometimes before the reference (line85: individuals.[13]) or after the reference (line 81: countries [12].) Check it through and make it for all references in the one or the other way. All the ref have been put before the “.”
• In case of:  .[24],[23][25] use [24-25], or [27][28][29][30][31] use [27-31].
• [57][55][58][43]. Consider the order 43,55,57,58. Also for other references like [2][65][34].

 Thanks, we will synthetize the reference in a more succinct way on the final draft. The references are managed now with a software that does not “contract” then in a group eg “[24-28]”, so this work will be done on the final draft, not to lose the information from the automatic listing. Same for the “order” of the references.

 

Table 1: would be nice to have it all on one page? Thanks for noting, we put now the table in one page.

 

Line 71-73: This sentence is very difficult to understand. HTA itself (because of missing best information) does not invest in low-value care or yield in poor population health outcomes. HTA may reveal that best information is missing and the technology, when used and reimbursed is of no value.

This is a crucial point on this dissertation. Some discrepant results from HTA exercises are based on large assumptions, therefore the final outputs across different HTA expert groups can be sometimes very different – even opposite. This is for example common in cost-effectiveness estimates across countries. The message we want to give is that poor data availability can implicate major disruption of the HTA process, including for the need of rough assumptions, that can lead to misleading interpretations or results, with “informed” uptake of poor value interventions. This is the opposite of what HTA should do, and is affected by an original, primordial issue of the poor data. We agree that the sentence sounds not proper, and rephrased as suggested by the expert reviewer.

 

Line 85-87: wasteful spending itself does not impact cancer outcomes….. it’s the non effective drugs that do so?

This is a great point, thanks for highlighting. Literature on this is poor, but overall seems to suggest that low-value care exposed patients to higher out of pocket expenditure, that is prognostic for patients with cancer. However, we agree that the real driver is poor-value drugs, not waste of resources itself. The sentence has been amended, accordingly.

 

Line 92-94: HTA itself cannot readjust the trajectory if low-value interventions. That can only be done by the health policy makers. They need to decide, based on the outcomes of a HTA, if they want to lessen wasteful spending and remove low-value interventions from a health program.

Agree with this, we rephrased the sentence to better outline the process, explicitly, and highlight the potential effect of HTA on health as a political choice, to inform – not itself redirect – decision-makers toward high-value.

Line 133. Remove the “.” after implication. Thanks for the careful assessment, this was removed.

Figure 2 (and 3) is of a bad graphical quality, try to improve. Thanks for highlighting it. We submitted high-resolution figures (vectorial). This Journal requires the pictures to be embedded in the main manuscript (lower quality) for the reviewers AND submitted separately as high quality.

Line 151: “for example before market» unclear what you mean. This was meant to address and explain the lifecycle-approach of HTA to products. We rephrased to make it clear.

 

Line 184: “direct clinical endpoints.» what do you mean by direct clinical endpoints? Please give some examples like overall survival. Thanks, this is very important to stress. Gold Standard endpoints are sometimes referred as “direct”, either for clinical trial designs, regulators and HTA, to underline they measure “directly” patient-relevant outcomes. The reviewer’s comment has been implemented.

Line 195-197. “Clinically meaningful improvement” I do agree that “OS” belongs to this, however many oncologist will not agree that PFS is NOT a clinically meaningful improvement! Being progression free can be highly important to patients and their awareness of their health state.

Thanks for the point, this is an intricate discussion, we agree. OS is still considered a gold standard, direct endpoint, along with QoL. While PFS is important for accelerated approvals, it is widely agreed and recognized that PFS alone cannot give a sense of the ultimate benefit of a new treatment. For example, recently, FDA withdraw umbralisib, an inhibitor of PIK3-delta, for the indication of indolent lymphomas. The accelerated approval was based on the ORR surrogate metric, then confirmed in large ph3 RCTs. The ph3 controlled trials confirmed better PFS (doubled!), but showed worse toxicities, more toxic deaths with an ultimate detriment on OS. PIK3-delta inhibitors have now been withdrawn as a class for that indication, because it is still unacceptable to disregard OS and QoL, despite large PFS benefits. This is how the interpretation of the importance to have surrogate endpoints “correlated” with direct endpoints should be done: make sure that ultimate OS and QoL outcomes are not negatively impacted – that can happen, and is not uncommon, despite large PFS gains. The sentence has been rephrased, and softened.

 

Line 216-217: do not use “Also” twice in the same sentence. And do you mean “whether or not”?

Thanks for the note, the sentence has been rephrased accordingly.

 

Line 319 “regardless of TMB high» should be “regardless of high TMB”?

Yes, agree, this is what is meant. The sentence has been amended.

 

Line 339 and 340: “deliver effective and sustainable HTA decisions” in my understanding HTA does not deliver decisions. It delivers the basis for policy makers to make “a decision”.  I would not write “One contributor to the lack of adequate data to inform HTA decisions” but for example “One contributor to the lack of adequate data to inform HTA outcomes” or HTA results.

Thanks for the comment, this is really sophisticated. We agree that this is about HTA outcomes to inform decisions makers. Rephrased accordingly.

 

Line 365-366: Although this is true, HTA can only be as good as the evidence which is used, so if clinical trials and approval of drugs stay on the same level as it is now, harmonising HTA requirements won’t solve the problem….

Thanks for comment, we agree harmonization is not an ultimate solution, but is a way to enhance capacity (by pooling national efforts and capacities) and agree on quality methodologies. We rephrased the entire paragraph to assure the nuanced implications are better captured.

 

Line 412-417: I do not understand what risk-sharing and outcome-based agreements have to do with performing a HTA. These are deals between health insurance, government and pharmaceutical companies and should not influence the production of a HTA. The whole paragraph somehow seems to mix up things, putting HTA in the context of performance-based systems. HTA should be an independent process of evaluating all possible outcomes, completely independent of what agreements on pricing or made.

Thanks for this point, I think complexity was generated from how HTA works in countries. In EU, some HTA agencies are embedded under the regulatory agencies and inform price decisions and participate to risk-sharing infrastructure.

 

Line 434-440: Again, I would not use “HTA decision”. It’s not the HTA who makes a decision. Again, use HTA Outcome. Thanks for the comment, this was implemented.

Reviewer 3 Report

REVIEW OF PAPER: “CHALLENGES FOR CONDUCTING HEALTH TECHNOLOGY ASSESSMENT FOR CANCER MEDICINES: THE CENTRAL PROBLEM OF SAFETY-EFFICACY UNCERTAINTY”

Journal: Current Oncology

 

Regarding the review of the paper "Challenges for conducting Health Technology Assessment for cancer medicines: the central problem of safety-efficacy uncertainty", the paper deals with a topic of interest and on which there is a large amount of recent literature, as it is a current issue: the approval of drugs for the treatment of cancer on the basis of what the authors call endpoints, which require validation processes that do not always offer sufficient guarantee at the end of the process in terms of the added value of the drugs approved for sale and distribution. on the other hand, time is sometimes wasted waiting for the survival results of the evaluations carried out on these drugs.

For the authors of this paper, the lack of data on drug efficacy leads to the adoption of treatments that may not add sufficient therapeutic value to patients.

Title and summary. The title and abstract express well the object of study, objectives, and results of the article.

Structure of the article: The contents are well organized and it includes a theoretical framework of the research problem but at this point, I suggest the authors incorporate some other bibliographic references that I miss in the text:

Oriana Ciani, Marc Buyse, Michael Drummond, Guido Rasi, Everardo D. Saad, Rod S. Taylor, Time to Review the Role of Surrogate End Points in Health Policy: State of the Art and the Way Forward, Value in Health, Volume 20, Issue 3, 2017, Pages 487-495

 Fleming TR, Powers JH. Biomarkers and surrogate endpoints in clinical trials. Stat Med. 2012 Nov 10;31(25):2973-84. doi: 10.1002/sim.5403. Epub 2012 Jun 18. PMID: 22711298; PMCID: PMC3551627.

Focusing on the opportunity of the study, it must be said that it is useful to work since it covers one of the major problems related to the sustainability, access, and efficiency of healthcare systems.

Materials and methods.

Regarding the material and methods section, the methodology is tailored to the object of study and the objectives and is explained in a transparent manner while it has been validly applied to guarantee the paper's development.

However, when using a literature review, it is suggested that authors include a table with the search strategy in the Material and Methods section,

Development of the paper.

In the development of the paper, it would be interesting for the authors to include the Flow diagram of the paper selection process, for better information and orientation of the readers.

 The authors' approach is interesting but should be contrasted more, in my opinion, with those who value the time that can be saved by making decisions with properly validated endpoints without waiting for robust evidence on survival as a consequence of treatment.

In this regard, I would like to propose some bibliographical references that the authors could discuss in their article:

Brooks N, Campone M, Paddock S, Shortenhaus S, Grainger D, Zummo J, Thomas S, Li R. Approving cancer treatments based on endpoints other than overall survival: an analysis of historical data using the PACE Continuous Innovation Indicators™ (CII). Drugs Context. 2017 Nov 15;6:212507. doi: 10.7573/dic.212507. PMID: 29167693; PMCID: PMC5699106.

Agyeman Abena S., Siegel Jeffrey N., Leptak Christopher. Establishing a Public Resource for Acceptable Surrogate Endpoints to Support FDA Marketing Applications . Frontiers in Medicine     VOLUME 9; 2022;

URL=https://www.frontiersin.org/article/10.3389/fmed.2022.820990     DOI=10.3389/fmed.2022.820990 

Schuster Bruce C, Brhlikova P, Heath J, McGettigan P. The use of validated and nonvalidated surrogate endpoints in two European Medicines Agency expedited approval pathways: A cross-sectional study of products authorised 2011-2018. PLoS Med. 2019 Sep 10;16(9):e1002873. doi: 10.1371/journal.pmed.1002873. PMID: 31504034; PMCID: PMC6736244.

Bibliography.

75% of the bibliography cited in the study belongs to the previous five years.

 

Overall, it is an interesting study and should be considered for publication in Current Oncology, once the minor revision proposed has been resolved.

Author Response

Reviewer 3

Regarding the review of the paper "Challenges for conducting Health Technology Assessment for cancer medicines: the central problem of safety-efficacy uncertainty", the paper deals with a topic of interest and on which there is a large amount of recent literature, as it is a current issue: the approval of drugs for the treatment of cancer on the basis of what the authors call endpoints, which require validation processes that do not always offer sufficient guarantee at the end of the process in terms of the added value of the drugs approved for sale and distribution. On the other hand, time is sometimes wasted waiting for the survival results of the evaluations carried out on these drugs. For the authors of this paper, the lack of data on drug efficacy leads to the adoption of treatments that may not add sufficient therapeutic value to patients.

We thank the reviewer for the careful assessment of our paper.

Title and summary. The title and abstract express well the object of study, objectives, and results of the article.

Structure of the article: The contents are well organized and it includes a theoretical framework of the research problem but at this point, I suggest the authors incorporate some other bibliographic references that I miss in the text:

Oriana Ciani, Marc Buyse, Michael Drummond, Guido Rasi, Everardo D. Saad, Rod S. Taylor, Time to Review the Role of Surrogate End Points in Health Policy: State of the Art and the Way Forward, Value in Health, Volume 20, Issue 3, 2017, Pages 487-495

 Fleming TR, Powers JH. Biomarkers and surrogate endpoints in clinical trials. Stat Med. 2012 Nov 10;31(25):2973-84. doi: 10.1002/sim.5403. Epub 2012 Jun 18. PMID: 22711298; PMCID: PMC3551627.

We thank the reviewer for the additional references. All of them were appropriately incorporated.

 

Focusing on the opportunity of the study, it must be said that it is useful to work since it covers one of the major problems related to the sustainability, access, and efficiency of healthcare systems.

This is so important and a critical message of our paper. Thanks for outlining clearly, we incorporated this powerful statement in the paper, in the intro para.

 

Materials and methods.

Regarding the material and methods section, the methodology is tailored to the object of study and the objectives and is explained in a transparent manner while it has been validly applied to guarantee the paper's development.

However, when using a literature review, it is suggested that authors include a table with the search strategy in the Material and Methods section.

Thanks for the comment, we expanded on that now. This is not a systematic review, and a scoping exercise was delivered in the preparation phase. Overall, it was developed as a narrative review, so PRISMA does not apply.

Development of the paper.

In the development of the paper, it would be interesting for the authors to include the Flow diagram of the paper selection process, for better information and orientation of the readers.

We agree that literature selection must be tracked in a systematic review. This journal requires to provide a methodology section also for narrative review. Therefore, we expanded on the methods but a flow chart would not be appropriate, as the approach was comprehensive, with an original research strategy implemented across the datasets, but with no systematic selection, inclusion/exclusion. Se also reply to reviewer 1 on this.

The authors' approach is interesting but should be contrasted more, in my opinion, with those who value the time that can be saved by making decisions with properly validated endpoints without waiting for robust evidence on survival as a consequence of treatment.

In this regard, I would like to propose some bibliographical references that the authors could discuss in their article:

Brooks N, Campone M, Paddock S, Shortenhaus S, Grainger D, Zummo J, Thomas S, Li R. Approving cancer treatments based on endpoints other than overall survival: an analysis of historical data using the PACE Continuous Innovation Indicators™ (CII). Drugs Context. 2017 Nov 15;6:212507. doi: 10.7573/dic.212507. PMID: 29167693; PMCID: PMC5699106.

Agyeman Abena S., Siegel Jeffrey N., Leptak Christopher. Establishing a Public Resource for Acceptable Surrogate Endpoints to Support FDA Marketing Applications . Frontiers in Medicine     VOLUME 9; 2022;URL=https://www.frontiersin.org/article/10.3389/fmed.2022.820990     DOI=10.3389/fmed.2022.820990 

Schuster Bruce C, Brhlikova P, Heath J, McGettigan P. The use of validated and nonvalidated surrogate endpoints in two European Medicines Agency expedited approval pathways: A cross-sectional study of products authorised 2011-2018. PLoS Med. 2019 Sep 10;16(9):e1002873. doi: 10.1371/journal.pmed.1002873. PMID: 31504034; PMCID: PMC6736244.

The references suggested were deemed all appropriate and valuable, therefore included in the manuscript. A contrast para was implemented, to show alternative opinions: this is very important in framing the comprehensive dialogue on the topic.

 

Bibliography.

75% of the bibliography cited in the study belongs to the previous five years.

Thanks for this check, we tried to report the most updated references.

 

Overall, it is an interesting study and should be considered for publication in Current Oncology, once the minor revision proposed has been resolved.

Thanks for your comprehensive and accurate revision.

Round 2

Reviewer 1 Report

I thank the authors for revising the manuscript based on the reviewers suggestions. Although improved, I think further refinement is required as the scope/objective of the review, and rationale for such, could be more clearly articulated. Additionally, although a review, there appears to be limited reviewed literature on the actual implications of discussed data/trial limitations on the HTA process. Specific examples included of data/trial limitations often pertain to regulatory approval (ex. FDA) which is not the same as HTA bodies. This is an important and interesting topic; however, I would recommend the authors to revise this manuscript to improve the focus of the review. 

Minor comments below:

1. Title - consider rephrasing to: "Implications of oncology trial design and uncertainties in primary data on health technology assessments"

2. Table 1 - What does "advisory or informing sharing role in some countries" refer to in the last row, last column?

3. Methods - was the search conducted with an expert librarian? Was it limited to English only? What were the dates of the search? It is added that "scoping research" was conducted - was this done in compliance with PRISMA reporting or JBI? The authors also include the results of the scoping review in the "methods" section. This should be expanded upon to ensure transparency that the topic was not "cherry-picked". This was also help with the next comment as it is not clear how data availability was chosen as the priority for this review. Although this is a narrative review, referring to the scoping research that was done to select this topic requires further elaboration which could be included in an appendix. 

4. Figure 1 - it is still unclear to me how the bar graph in Figure 1 was constructed. The included link also does not link to a functioning webpage.

5. 2.1 Defining HTA - health technology also encompasses non-systemic therapies (i.e. radiation therapy, surgical approaches, etc). 

6. References are limited throughout the manuscript - as a review, this paper should refer to the reviewed literature. In the absence of this, it appears more as a commentary. 

7. Section 2 may not be required or should be summarized in the introduction if the intent of this review is to look at how clinical trial design and data uncertainties impact HTA.

8. Section 3.1.4 - the authors describe the issues with early approval; however, as this paper is focused on impact on HTA, the discussion is best directed at examples which impacted HTA processes. The examples discussed speak of treatments that were granted regulatory approval but not necessarily positive recommendation through formalized HTA bodies. 

9. Section 4.2 - I think the major challenge with tumor agnostic trials is the lack of a comparator arm for each of the tumor types. Often tumor response is reported by tumor type. Consider adjusting this section. 

10. Section 6.4.1 - I do not see a clear link between choosing wisely and HTA of cancer medicines. If the authors wish to keep this section, further elaboration is required. 

Author Response

1. Title - consider rephrasing to: "Implications of oncology trial design and uncertainties in primary data on health technology assessments".

We thank the expert reviewer for the input. The title has been amended to better mirror the contents, as suggested.

 

2. Table 1 - What does "advisory or informing sharing role in some countries" refer to in the last row, last column?

Amended, thanks.

 

3. Methods - was the search conducted with an expert librarian? Was it limited to English only? What were the dates of the search? It is added that "scoping research" was conducted - was this done in compliance with PRISMA reporting or JBI? The authors also include the results of the scoping review in the "methods" section. This should be expanded upon to ensure transparency that the topic was not "cherry-picked". This was also help with the next comment as it is not clear how data availability was chosen as the priority for this review. Although this is a narrative review, referring to the scoping research that was done to select this topic requires further elaboration which could be included in an appendix. 

Dear reviewer, in the methods we wrote “This paper presents a review of the challenges in conducting HTA for cancer medicines arising from oncology trial designs and uncertainties of safety-efficacy data. Methods: Multiple databases (PubMed, Scopus and Google Scholar) and grey literature (public health agencies and governmental reports) were searched to inform this policy narrative review”. It is clear this is a narrative review, the methodology section is required by this Journal, albeit it is not standard for narrative reviews in all the journals.

The initial scoping review was provided to understand the landscape and evaluate the up-to-date literature, however it is incorrect to say that the literature was cherry-picked in some instrumental way, or non-transparent. The panel of authors is diverse with expertise in HTA or contributing to it, from different countries.

As a narrative review, the PRISMA does not apply, and a librarian was not involved in the research of the literature.

This is now specified also in section 1.1.

 

4. Figure 1 - it is still unclear to me how the bar graph in Figure 1 was constructed. The included link also does not link to a functioning webpage.

Thanks for the comment. The data source, as specified, is the WHO repository for HTA, specifically a recent global survey on the topic available at https://www.who.int/teams/health-systems-governance-and-financing/economic-analysis/health-technology-assessment-and-benefit-package-design/survey-homepage . At this link, find the data at “Click here to access the Global HTA & HBP Summary Results Dashboard” to access the full datased. Then go on “barriers to HTA”,  and you will see displayed the data of the “barriers” according to the income. Hopefully this is enough clear now. The Report is referenced in the bibliography and also mentioned in the caption of Figure 1.

 

5. 2.1 Defining HTA - health technology also encompasses non-systemic therapies (i.e. radiation therapy, surgical approaches, etc). 

Thanks for this comment, this is a paper on cancer medicines. The broader applications of HTA to other health interventions is reported in paragraph 2.1 “In cancer, health technology encompasses a broad range of health interventions, including therapeutics, ranging from small-molecule medicines, complex biologicals, to cell- and gene-based therapies.”.

 

6. References are limited throughout the manuscript - as a review, this paper should refer to the reviewed literature. In the absence of this, it appears more as a commentary. 

We included 111 distinct references: according to the guidelines of this Journal, we tried to limit the choice to the most updated references of the last 5 years. This Journal has rules in term of length of manuscript and N of references. Many journals limit the N of max references to 50-75, we are above that and feel the reviewed literature is appropriate to inform the discussion.

7. Section 2 may not be required or should be summarized in the introduction if the intent of this review is to look at how clinical trial design and data uncertainties impact HTA.

As authors, we do believe there is need for this section, based on the purposes of this Journal. This is a Journal for clinicians, not only policy makers. Therefore, there is need to give more elements of background, because not all oncologists are trained in HTA and broadly, cancer policy. Please to be cognizant of this aspect.

 

8. Section 3.1.4 - the authors describe the issues with early approval; however, as this paper is focused on impact on HTA, the discussion is best directed at examples which impacted HTA processes. The examples discussed speak of treatments that were granted regulatory approval but not necessarily positive recommendation through formalized HTA bodies. 

This is correct, but the implications on poor safety-efficacy data to inform HTA have been articulated in the paper. The main focus of this manuscript is not about the ultimate recommendations of HTA, that may be based on assumptions when data are limited, and local resource constraints, but on the implications of poor data on conducting HTA. We did not on purpose comment on HTA outcomes, in term of recommendations, because they are very different across institutions, especially when data are limited and assumptions are large.

 

 

9. Section 4.2 - I think the major challenge with tumor agnostic trials is the lack of a comparator arm for each of the tumor types. Often tumor response is reported by tumor type. Consider adjusting this section. 

Thanks, this is noted. We articulated on the lack of comparators and the limits by grouping several tumours together. We included the specification of this reviewer in term of tumor-specific outcomes.

 

10. Section 6.4.1 - I do not see a clear link between choosing wisely and HTA of cancer medicines. If the authors wish to keep this section, further elaboration is required. 

This was discussed and agreed across the authors. The Choose Wisely framework is now included broadly in the dialogue around value-based priority setting, and we feel it is important to acknowledge. We did elaborate the paragraph.

 

Reviewer 2 Report

Thanks a lot for the reviced version, which is in my opinion corrected well.

Author Response

We thank the expert reviewer for the accurate inputs that improved greatly our manuscript and indeed we do appreciate the efforts and quick reply to our revisions.