Rebecca Trowman, Antonio Migliore, Daniel A Ollendorf
{"title":"Designing collaborations involving health technology assessment: discussions and recommendations from the 2024 health technology assessment international global policy forum.","authors":"Rebecca Trowman, Antonio Migliore, Daniel A Ollendorf","doi":"10.1017/S0266462324000436","DOIUrl":"10.1017/S0266462324000436","url":null,"abstract":"<p><p>Although collaboration is an intensive way of working together, it is essential for such efforts to achieve shared goals. Health technology assessment (HTA) is transdisciplinary and has an important history of collaboration, with collaboration featuring increasingly in the strategic plans of HTA bodies and stakeholders. Collaboration can be between HTA bodies and between HTA bodies and other stakeholders-most notably regulators but increasingly payers, patient and caregiver organizations, clinicians-clinical societies, and academia. The 2024 HTAi Global Policy Forum (GPF) discussed collaborations involving HTA bodies, reviewing existing and previous collaborations to see what has worked and what can be learned. Core discussion themes included: (i) determining the collaboration purpose is essential but may be dynamic, changing over time; (ii) choosing the collaboration topic takes time, requiring upfront investment and stakeholder mapping; (iii) inviting the right participants and treating them equally is important, including those who can impact HTA, those who will be impacted by HTA and those who bring new information; (iv) collaborations need clear governance, defined roles, responsibilities, metrics, and case study-pilots can be a useful operational model; (v) resourcing collaborations sustainably is a challenge-the time, people, and money required are often under-estimated; (vi) undertaking continual, iterative learning reviews ensures ongoing value and impact of collaborations. Recommendations for future work include the development of a \"go/no-go\" checklist to determine when collaboration is needed, supplemented with a set of \"best practice\" principles for establishing and working in collaborations involving HTA bodies.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e41"},"PeriodicalIF":4.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11563176/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568457","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jamie Elvidge, Nick Crabb, Diana Delnoij, Saskia Knies, Douglas Lundin, François Houÿez, Juha Röning, Junfeng Wang, Li Jiu, Alastair Bennett, Yingying Zhang, Dalia Dawoud
{"title":"Implementing a sandbox approach in health technology assessment: benefits and recommendations.","authors":"Jamie Elvidge, Nick Crabb, Diana Delnoij, Saskia Knies, Douglas Lundin, François Houÿez, Juha Röning, Junfeng Wang, Li Jiu, Alastair Bennett, Yingying Zhang, Dalia Dawoud","doi":"10.1017/S0266462324000412","DOIUrl":"10.1017/S0266462324000412","url":null,"abstract":"<p><strong>Objectives: </strong>The sandbox approach, developed in the financial technologies sector, creates an environment to collaboratively develop and test innovative new products, methods and regulatory approaches, separated from business as usual. It has been used in health care to encourage innovation in response to emerging challenges, but, until recently, has not been used in health technology assessment (HTA). This article summarizes our learnings from using the sandbox approach to address three challenges facing HTA organizations and to identify implications for the use of this approach in HTA.</p><p><strong>Methods: </strong>We identified three challenging contemporary HTA-related topics to explore in a sandbox environment, away from the pressures and interests of \"live\" assessments. We convened a pool of 120 stakeholders and experts to participate in various sandbox activities and ultimately co-develop solutions to help HTA organizations respond to the identified challenges.</p><p><strong>Results: </strong>Important general learnings about the potential benefits and implementation of a sandbox approach in HTA were identified. Consequently, we developed recommendations to guide its use, including how to implement an HTA sandbox in an effective way and the types of challenges for which it may be best suited.</p><p><strong>Conclusions: </strong>For many HTA organizations, it is difficult to carefully consider emerging challenges and innovate their processes due to risks associated with decision errors and resource limitations. The sandbox approach could reduce these barriers. The potential benefits of addressing HTA challenges in a collaborative \"safe space\" are considerable.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e44"},"PeriodicalIF":2.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11569896/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568478","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The newcomer's guide to health technology assessment: a collection of resources for early career professionals.","authors":"Antonio Migliore, Debjani Mueller, Wija Oortwijn","doi":"10.1017/S0266462324000424","DOIUrl":"10.1017/S0266462324000424","url":null,"abstract":"","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e38"},"PeriodicalIF":4.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11563173/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568482","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pietro Refolo, Katherine Duthie, Björn Hofmann, Michal Stanak, Neil Bertelsen, Bart Bloemen, Rosella Di Bidino, Wija Oortwijn, Costanza Raimondi, Dario Sacchini, Gert Jan van der Wilt, Kenneth Bond
{"title":"Ethical challenges for Health Technology Assessment (HTA) in the evolving evidence landscape.","authors":"Pietro Refolo, Katherine Duthie, Björn Hofmann, Michal Stanak, Neil Bertelsen, Bart Bloemen, Rosella Di Bidino, Wija Oortwijn, Costanza Raimondi, Dario Sacchini, Gert Jan van der Wilt, Kenneth Bond","doi":"10.1017/S0266462324000394","DOIUrl":"10.1017/S0266462324000394","url":null,"abstract":"<p><p>Since its inception, Health Technology Assessment (HTA) has typically determined the value of a technology by collecting information derived from randomized clinical trials (RCTs), in line with the principles of evidence-based medicine (EBM). However, data from RCTs did not constitute the sole source of information, as other types of evidence (such as primary qualitative research) have often been utilized. Recent advances in both generating and collecting other types of evidence are broadening the landscape of evidence, adding complexity to the discussion of \"robustness of evidence.\" What are the consequences of these recent developments for the methodology and conduct of HTA, the HTA community, and its ethical commitments? The aim of this article is to explore some ethical challenges that are emerging in the current evolving evidence landscape, particularly changes in evidence generation and collection (e.g., diversification of data sources), and shifting standards of evidence in the field of HTA (e.g., increasing acceptability of evidence that is thought of as lower quality). Our conclusion is that deciding how to best maintain trustworthiness is common to all these issues.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e39"},"PeriodicalIF":2.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11569911/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568462","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ryan Rachmad Nugraha, Christian Suharlim, Rozar Prawiranegara, Arry Lesmana Putra, Mutia A Sayekti, Armansyah Armansyah, Lusiana Siti Masytoh, Sweta Saxena, Anastasia Susanto, John C Langerbrunner, Nurul Maretia Rahmayanti, Miyoung Choi, Budi Wiweko, Budi Hidayat
{"title":"A framework for improved collaboration on HTA in the Asia-Pacific region: a role for HTAsiaLink.","authors":"Ryan Rachmad Nugraha, Christian Suharlim, Rozar Prawiranegara, Arry Lesmana Putra, Mutia A Sayekti, Armansyah Armansyah, Lusiana Siti Masytoh, Sweta Saxena, Anastasia Susanto, John C Langerbrunner, Nurul Maretia Rahmayanti, Miyoung Choi, Budi Wiweko, Budi Hidayat","doi":"10.1017/S0266462324000588","DOIUrl":"10.1017/S0266462324000588","url":null,"abstract":"<p><p>Countries frequently use health technology assessment (HTA) to set priorities for introducing new interventions or evaluating existing interventions; however, applying the tool effectively is heavily dependent on a country's resources and capacity. Infrastructure and data, technical expertise, broad stakeholder involvement, and financial support are required to improve HTA processes. In the Asia-Pacific, HTAsiaLink was established to facilitate this practice, but strengthening and legitimizing this organization are needed to maximize its potential to support HTA institutionalization in the region. To realize this objective, HTAsiaLink can serve as a center of excellence while providing experiential learning and sharing information. As a learning hub, HTAsiaLink can share resources-particularly data-that can contribute to joint HTAs as done in the European Union and strengthen capacity in countries needing to develop their HTA expertise.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e43"},"PeriodicalIF":4.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11563177/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568452","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Denny John, Abhirup Dutta Majumdar, Rakesh N Pillai, Sajda Khatoon, Paramita Bhattacharya, Nirmalya Mukherjee, Jaideep C Menon, K Pavithran, Anand Kumar, Mihajlo Jakovljevic
{"title":"Health technology assessment for digital health Technologies in India: a framework for action.","authors":"Denny John, Abhirup Dutta Majumdar, Rakesh N Pillai, Sajda Khatoon, Paramita Bhattacharya, Nirmalya Mukherjee, Jaideep C Menon, K Pavithran, Anand Kumar, Mihajlo Jakovljevic","doi":"10.1017/S0266462324000345","DOIUrl":"10.1017/S0266462324000345","url":null,"abstract":"<p><strong>Objectives: </strong>The aim of this study is to understand the path for establishing digital health technologies-health technology assessment (DHT-HTA) in India.</p><p><strong>Methods: </strong>A rapid review of HTA and DHT frameworks on PubMed (MEDLINE) and Google Scholar was conducted to identify DHT-HTA guidelines, and HTA processes in India. MS-Excel template was created with key domains for assessing DHT in resource-constrained settings based on studies and reports identified. Responses received from seventeen experts with varying expertise in DHT, HTA, clinical, and research were contacted using an online form. Following the principles of qualitative research rooted on grounded theory approach, themes and domains were derived for a framework which was again circulated through participants. Weightage for each theme was assigned based on the frequency of responses and qualifiers were used to interpret results. Inductively derived themes from these responses were clubbed together to identify macro-level systems requirements, and finally pre-requisites for setting up DHT-HTA framework was synthesized.</p><p><strong>Results: </strong>HT are commonly perceived by experts (64.7 percent participants) as a technology strictly connected to health information. Real-world data (i.e., electronic health data) are recognized as a relevant tool in support of decision-making for clinical and managerial levels. Experts identified some pre-requisites for the establishment of DHT-HTA in the country in terms of infrastructure, contextual factors, training, finance, data security, and scale-up.</p><p><strong>Conclusion: </strong>Our research not only identified the pre-requisites for the adoption of a DHT-HTA framework for India, but confirmed the need to address DHT-HTA's acceptability among. Hospitals and health insurance providers.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":" ","pages":"e70"},"PeriodicalIF":2.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568448","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Santiago Hasdeu, Anabel Beliera, Jorgelina Alvarez, Julián Sánchez Viamonte
{"title":"Exploring the linkage between health technology assessment and decision making during COVID-19 public health emergency in a developing country: analysis of processes and results.","authors":"Santiago Hasdeu, Anabel Beliera, Jorgelina Alvarez, Julián Sánchez Viamonte","doi":"10.1017/S0266462324000473","DOIUrl":"10.1017/S0266462324000473","url":null,"abstract":"<p><strong>Objectives: </strong>The decision-making (DM) process in public administration is the subject of research from different perspectives and disciplines. Evidence-based policies, such as health technology assessment (HTA), are not the only support on which public policies are designed. During the COVID-19 pandemic WHO, national and subnational institutions developed HTA reports to guide DM. Despite this, inadequate variability was observed in the health technologies recommended and reimbursed by different provincial Health Ministries in a federally organized developing country like Argentina. The processes and results of DM on health technologies for COVID-19 in Health Ministries of Argentina were inquired.</p><p><strong>Methods: </strong>A retrospective research design was developed, with triangulation of quantitative and qualitative methods. We retrieved information for the years 2020-2021 through document review of official webpages, surveys, and interviews with decision-makers of the 25 Argentinian Ministries of Health. We analyzed the recommendations and reimbursement policies of seven health technologies.</p><p><strong>Results: </strong>In contradiction with WHO's policies, ivermectine, inhaled ibuprofen, convalescent plasma and equine serum were widely recommended by most of Argentina's health ministries outside a clinical trial context, with risks for patients and a huge opportunity cost.</p><p><strong>Conclusions: </strong>Despite an important HTA institutional capacity, the impact of HTA organizations and their technical reports was limited. Health Ministries with institutionalized HTA units had more adherence to WHO recommendations, but the influence of different technical and political criteria was identified. Power relations within and outside the administration, the pharmaceutical industry and academics, the media, social pressure, the judicial and legislative powers, and the political context strongly influenced DM.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e42"},"PeriodicalIF":4.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11563179/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568468","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Health technology assessment of medicinal products in Greece: a 5-year (2018-2023) review of timelines and productivity.","authors":"Athanasios Chantzaras, Athanasios Margetis, Chara Kani, Vassilis Koutsiouris, Flora Bacopoulou","doi":"10.1017/S0266462324000485","DOIUrl":"10.1017/S0266462324000485","url":null,"abstract":"<p><strong>Objectives: </strong>To assess the health technology assessment (HTA) process in Greece from its establishment in 2018 until 2023 in terms of timeliness and productivity.</p><p><strong>Methods: </strong>Data were collected from the HTA Committee's database and other publicly available sources. The overall study timeframe was divided into three periods: (i) July 2018-January 2020, (ii) January 2020-July 2021, and (iii) July 2021-February 2023.</p><p><strong>Results: </strong>During the study period, a total of 1,157 applications for medicinal products (MPs) (including 219 new active substances (NAS) and orphans) were submitted to the HTA Committee. The number of HTA recommendations increased from 60 (first period) to 641 (third period), while the backlog of MPs pending HTA and price negotiations decreased from 89 and 106 (January 2020) to 8 and 44 (February 2023), respectively. The median time intervals for all application types decreased significantly over time. In February 2023, the median time for clinical data assessment of NAS (excluding orphans) almost halved from 207 days in the first period to 114 days; median times for NAS and orphans from regulatory approval to HTA application were 420 and 457 days, and from HTA application to reimbursement 228 and 417 days, respectively.</p><p><strong>Conclusions: </strong>The performance of the HTA process in Greece improved significantly over time, with increased MP appraisals, backlog reduction, and decreased timelines. Delays in reimbursement of NAS were mainly caused by the long gap between regulatory approval and HTA application. Overall, HTA review times in Greece are now on par with that of well-established European HTA systems.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e40"},"PeriodicalIF":4.6,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11563181/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568472","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Perceptions of patients and caregivers toward the management of rare disease in Malaysia: a qualitative research study.","authors":"Azuwana Supian, Asrul A Shafie, Lock-Hock Ngu, Hatijah Ayob, Nathorn Chaiyakunapruk","doi":"10.1017/S0266462324000333","DOIUrl":"10.1017/S0266462324000333","url":null,"abstract":"<p><strong>Objective: </strong>The management of rare diseases is rarely addressed among policymakers and public communities. It is hindered by the lack of information on its epidemiology and burden, especially from the perspective of patients and families with rare diseases. This study aims to understand the perceptions of rare disease patients and their families in the management of rare diseases in Malaysia.</p><p><strong>Methodology: </strong>A qualitative interview was used to explore the perceptions of patients and families regarding the management of rare diseases in Malaysia. In-depth interviews were conducted with the rare disease patients or their parents/guardians provided by three major rare disease advocacy groups, between 1 July and 15 September 2016. The interviews focused on two key areas: the challenges associated with rare disease and the issues related to accessing medication.</p><p><strong>Findings: </strong>Out of the nineteen recruited participants, seventeen (89.5 percent) completed the interview sessions. The significance of awareness, knowledge, and support from others emerged as crucial for families and patients living with rare diseases. Despite facing delayed diagnosis and treatment, a majority of patients and parents expressed satisfaction with the advancements in rare disease management. Nevertheless, a prominent challenge revolves around access to enzyme replacement therapy for eligible patients.</p><p><strong>Conclusion: </strong>This study emphasizes the importance of healthcare professionals understanding patient with rare diseases perceptions to tailor communication strategies, provide accurate information, and address concerns effectively. The message underscores the significance of collaboration between healthcare providers and patient support groups to deliver adequate health information, potentially enhancing patients' understanding and their illness perceptions.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e34"},"PeriodicalIF":2.6,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11569895/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142500554","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Linda Mundy, Ben Forrest, Li-Ying Huang, Guy Maddern
{"title":"Health technology assessment and innovation: here to help or hinder?","authors":"Linda Mundy, Ben Forrest, Li-Ying Huang, Guy Maddern","doi":"10.1017/S026646232400059X","DOIUrl":"10.1017/S026646232400059X","url":null,"abstract":"<p><p>Innovative health technologies offer much to patients, clinicians, and health systems. Policy makers can, however, be slow to embrace innovation for many reasons, including a less robust body of evidence, perceived high costs, and a fear that once technologies enter the health system, they will be difficult to remove. Health technology funding decisions are usually made after a rigorous health technology assessment (HTA) process, including a cost analysis. However, by focusing on therapeutic value and cost-savings, the traditional HTA framework often fails to capture innovation in the assessment process. How HTA defines, evaluates, and values innovation is currently inconsistent, and it is generally agreed that by explicitly defining innovation would recognize and reward and, in turn, stimulate, encourage, and incentivize future innovation in the system. To foster innovation in health technology, policy needs to be innovative and utilize other HTA tools to inform decision making including horizon scanning, multicriteria decision analysis, and funding mechanisms such as managed agreements and coverage with evidence development. When properly supported and incentivized, and by shifting the focus from cost to investment, innovation in health technology such as genomics, point-of-care testing, and digital health may deliver better patient outcomes. Industry and agency members of the Health Technology Assessment International Asia Policy Forum (APF) met in Taiwan in November 2023 to discuss the potential of HTA to foster innovation, especially in the Asia region. Discussions and presentations during the 2023 APF were informed by a background paper, which forms the basis of this paper.</p>","PeriodicalId":14467,"journal":{"name":"International Journal of Technology Assessment in Health Care","volume":"40 1","pages":"e37"},"PeriodicalIF":4.6,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11563175/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142500553","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}