Developing and testing an evaluation framework for climate services for adaptation

Abstract

Climate services are increasingly developed and used to plan for climate change adaptation, but their success is poorly evaluated. A main reason is that an operational framework to support climate service researchers and practitioners pursuing evaluation is lacking. This study addresses this gap by developing and testing a robust and systematic evaluation framework in three steps. First, we designed a framework by operationalising agreed upon criteria for assessing climate service success. Second, the framework was tested in two climate service cases. Third, the usability, credibility, and transparency of the framework was assessed by climate service researchers and practitioners, including those engaged in the cases.

Our findings show that developed framework offers a standardized approach to evaluation, providing indicators, metrics, and guidance that enable the evaluator to provide a quantitative rating for each criterion. However, the robustness of ratings in the two cases was compromised due to limited interaction with targeted users during the development process and lack a of clear goals set from the beginning. This hampered incorporating the perception of a representative group of users and measuring impacts. Overall, the framework was considered usable by researchers and practitioners for various applications, including using it as design criteria, to facilitate learning, to guide development, and to support monitoring and evaluation. While generally perceived as credible and transparent, the framework would benefit from further testing and elaboration into practical materials. The study highlights that evaluation is done best when evaluation criteria are considered early in the development of the climate service.

Practical implications

Climate services are seen as important means to support and accelerate adaptation action. While investments in climate service development and use are increasing, their evaluation typically falls short. One reason for this is the lack of a sound evaluation framework. This study aimed to develop a robust and systematic evaluation framework that can be used in both science and practice settings. The framework was tested in two implemented climate service cases, and evaluated by climate service users, practitioners, and researchers, as well as by the evaluators themselves. Supplementary file 2 provides the framework, and an accompanying protocol describing important process steps to apply it. It also offers guidance on how to consider the success criteria during the development stages of a climate service, through guiding questions and a checklist. Here we present the practical implications of this study by (1) outlining the basic principles of the framework, summarizing the results of (2) testing and (3) evaluating the framework that have most practical relevance, and (4) highlighting suggestions for improving evaluation practice.

1) Basic principles of the framework are:

• •
  It can be used for different types of evaluation (e.g. summative, formative, developmental) and applied to the broad range of possible climate services.
• •
  It is based on 12 success criteria selected in a Delphi study, where experts evaluated which elements are most relevant to define the success of climate services for adaptation (Boon et al., 2024). If deemed necessary for a specific climate service or context, criteria can be added.
• •
  It offers a total of 20 indicators with supporting metrics and directions to measure the criteria. Indicators were selected based on literature review, considering the most robust approach for measurement while dealing with time and budget restrictions.
• •
  Each criterion is evaluated on a scale from 1: unsuccessful to 5: successful, allowing easy comparison between climate services and monitoring over time. The robustness of the rating is assessed by considering the representativeness of the sample and the extent to which evidence was validated through multiple sources.

2) Testing the framework in two cases shows:

• •
  The framework was usable to evaluate the criteria consistently, supported by clear metrics and instructions for measurement.
• •
  Challenges emerged for evaluating those indicators that require a clear definition of targeted users and goals, and for those that are measured through user perception. In both cases, users and goals were described only in general terms, which made it difficult and sometimes impossible to measure results for these indicators. Furthermore, the robustness of many ratings was compromised due to the difficulty in accessing a representative group of targeted users.
• •
  The evaluation results, including identified learnings, were recognized and appreciated by the involved stakeholders.

3) Evaluating the evaluation framework by climate service users, practitioners, and researchers shows:

• •
  The framework was considered usable for various applications, such as including it in the terms of reference of calls for tenders, developing business models, using it as design criteria, guiding development processes, supporting monitoring and evaluation, and facilitating learning about what works and what doesn’t work.
• •
  It was considered credible and transparent, although it needs further testing in different types of services and contexts, and may require further development of easy-to-use evaluation materials.
• •
  Especially the climate service producers and practitioners valued the framework.

4) Suggestions for improving evaluation practice:

• •
  The study highlights once again that good evaluation is done best when it is integrated early in the development process of a climate service. This approach not only allows for efficient data collection, but also helps establish more robust ratings by clearly defining users and goals of the climate service and setting up user interaction channels. This may lead to more successful services.
• •
  To stimulate the uptake of the framework and foster a culture for evaluation we see two promising pathways: 1) promoting the use of the success criteria as a helpful tool to guide and structure the climate service development process. Increasing awareness of the criteria may pave the way for more systematic efforts to evaluate the services; 2) promoting the necessity for evaluation, for example to be able to mitigate misguided or ineffective services. This could be done through mandatory use and evaluation of the success criteria through design or reporting requirement by commissioning parties.