{"title":"Can we use deliberation to change evaluation systems? How an advisory group contributed to policy change","authors":"Peter Dahler-Larsen","doi":"10.1177/13563890231156955","DOIUrl":"https://doi.org/10.1177/13563890231156955","url":null,"abstract":"One of the most dominant trends in the field of evaluation in recent years is the institutionalization of evaluation under headlines such as “evaluation culture,” “evaluation policy,” and “evaluation systems.” There has been less interest in how evaluation systems can be changed, modified, or improved, not to mention deinstitutionalized, if necessary. Can a variety of stakeholders deliberate about the consequences of an evaluation system, and can it lead to policy change? A case study of a ministerial advisory group on national tests in Denmark shows how specific challenges were dealt with, such as the design of the deliberative process, the potential dominance of experts, and the distinction between technical-evaluative and practical-political arguments, and how these maneuvers paved the way for policy change. Based on the case study, the article discusses the prospects for democratic deliberation about evaluation systems.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2023-03-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73010667","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Deborah D. DiLiberto, C. Opondo, S. Staedke, Clare I. R. Chandler, E. Allen
{"title":"The use of mediation analysis in evaluation of complex health interventions","authors":"Deborah D. DiLiberto, C. Opondo, S. Staedke, Clare I. R. Chandler, E. Allen","doi":"10.1177/13563890221149452","DOIUrl":"https://doi.org/10.1177/13563890221149452","url":null,"abstract":"This article presents an application of the causal inference approach to mediation analysis using the example of a complex intervention that aimed to improve the quality of care at health centres in Uganda. Mediation analysis is a statistical method that aims to isolate the causal mechanisms that make an intervention work in a given context. We combined data from a cluster randomized control trial and a mixed-methods process evaluation. We developed two causal models following our hypotheses of how the intervention was intended to work through mechanisms at health centres to improve health outcomes in the community. In adjusted analyses, there was evidence of an effect of the intervention on some health centre mechanisms; however, these did not lead to improvements in community health outcomes. We discuss the practical and epistemological challenges encountered when using mediation analysis to evaluate a complex intervention. These findings will inform future evaluations. Trial registration: The trial reported in this article is registered at: clinicaltrials.gov, NCT01024426. Registered 2 December 2009, https://clinicaltrials.gov/ct2/show/record/NCT01024426?term=NCT01024426&draw=2&rank=1","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2023-02-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"75357772","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Editorial Message from the New Editor-in-Chief","authors":"","doi":"10.1016/S0166-5316(23)00007-X","DOIUrl":"https://doi.org/10.1016/S0166-5316(23)00007-X","url":null,"abstract":"","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49867195","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"French language abstracts","authors":"","doi":"10.1177/13563890221142105","DOIUrl":"https://doi.org/10.1177/13563890221142105","url":null,"abstract":"","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87710480","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Thi Thuy Nga Nguyen , Olivier Brun , Balakrishna J. Prabhu
{"title":"A learning-based scheme for channel allocation to vehicular users in wireless networks","authors":"Thi Thuy Nga Nguyen , Olivier Brun , Balakrishna J. Prabhu","doi":"10.1016/j.peva.2023.102331","DOIUrl":"https://doi.org/10.1016/j.peva.2023.102331","url":null,"abstract":"<div><p>Resource allocation algorithms in wireless networks can require solving complex optimization problems at every decision epoch. For large scale networks, when decisions need to be taken on time scales of milliseconds, using standard convex optimization solvers for computing the optimum can be a time-consuming affair that may impair real-time decision making. In this paper, we propose to use Data-driven and Deep Feedforward Neural Networks (DFNN) for learning the relation between the inputs and the outputs of two such resource allocation algorithms that were proposed in Nguyen et al. (2019, 2020). On numerical examples with realistic mobility patterns, we show that the learning algorithm yields an approximate yet satisfactory solution with much less computation time.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49901372","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Practical anonymization for data streams: z-anonymity and relation with k-anonymity","authors":"Nikhil Jha , Luca Vassio , Martino Trevisan , Emilio Leonardi , Marco Mellia","doi":"10.1016/j.peva.2022.102329","DOIUrl":"https://doi.org/10.1016/j.peva.2022.102329","url":null,"abstract":"<div><p>With the advent of big data and the emergence of data markets, preserving individuals’ privacy has become of utmost importance. The classical response to this need is anonymization, i.e., sanitizing the information that, directly or indirectly, can allow users’ re-identification. Among the various approaches, <span><math><mi>k</mi></math></span>-anonymity provides a simple and easy-to-understand protection. However, <span><math><mi>k</mi></math></span>-anonymity is challenging to achieve in a continuous stream of data and scales poorly when the number of attributes becomes high.</p><p>In this paper, we study a novel anonymization property called <span><math><mi>z</mi></math></span>-anonymity that we explicitly design to deal with data streams, i.e., where the decision to publish a given attribute (atomic information) is made in real time. The idea at the base of <span><math><mi>z</mi></math></span>-anonymity is to release such attribute about a user only if at least <span><math><mrow><mi>z</mi><mo>−</mo><mn>1</mn></mrow></math></span> other users have exposed the same attribute in a past time window. Depending on the value of <span><math><mi>z</mi></math></span>, the output stream results <span><math><mi>k</mi></math></span>-anonymized with a certain probability. To this end, we present a probabilistic model to map the <span><math><mi>z</mi></math></span>-anonymity into the <span><math><mi>k</mi></math></span>-anonymity property. The model is not only helpful in studying the <span><math><mi>z</mi></math></span>-anonymity property, but also general enough to evaluate the probability of achieving <span><math><mi>k</mi></math></span>-anonymity in data streams, resulting in a generic contribution.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49901370","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Konstantin Avrachenkov , Evsey Morozov , Ruslana Nekrasova
{"title":"Stability analysis of two-class retrial systems with constant retrial rates and general service times","authors":"Konstantin Avrachenkov , Evsey Morozov , Ruslana Nekrasova","doi":"10.1016/j.peva.2022.102330","DOIUrl":"https://doi.org/10.1016/j.peva.2022.102330","url":null,"abstract":"<div><p>We establish stability criterion for a two-class retrial system with Poisson inputs, general class-dependent service times and class-dependent constant retrial rates. We also characterize an interesting phenomenon of partial stability when one orbit is tight but the other orbit goes to infinity in probability. All theoretical results are illustrated by numerical experiments.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49901371","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Combining process tracing and synthetic control method: Bridging two ways of making causal inference in evaluation research","authors":"F. Podestà","doi":"10.1177/13563890221139511","DOIUrl":"https://doi.org/10.1177/13563890221139511","url":null,"abstract":"This article discusses potential ways of combining two methods of evaluation in single-case studies: the synthetic control method and the process tracing method. Both are designed to examine certain events/programmes that take place in given cases but view these events/programmes from different causal perspectives. Seeing an event/programme as a cause, synthetic control estimates its impact on one or more outcomes. Conversely, starting from a certain outcome, process tracing uncovers the causes responsible. One can start from the causal explanation reached via one of the two methods and then proceed to examine that explanation through the other method. Once the causes of an outcome are traced via a process tracing analysis, that account can be validated by estimating the effects of those causes via synthetic control. Equally, once the impact of a certain event is estimated through synthetic control, causal mechanisms traceable via process tracing can be exploited to refine that impact evaluation.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80939793","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Evaluating the impact of business coaching programmes by taking perceptions seriously","authors":"G. Ton, F. van Rijn, H. Pamuk","doi":"10.1177/13563890221137611","DOIUrl":"https://doi.org/10.1177/13563890221137611","url":null,"abstract":"The paper addresses the challenges of evaluating the impact of business coaching programmes with a varied portfolio of firms working across sectors and countries. Observable indicators of changes in business management practices are rarely relevant across sectors. Therefore, evaluators need to rely on the perceptions of the managers who have received coaching. We designed an online survey to compare the effectiveness of business coaching within a portfolio and across programmes. The survey was applied to the portfolio of two private sector development programmes. We derived so-called ‘contribution scores’ from individuals’ perceptions of how business management practices had changed and their perceptions of the role of business coaching in bringing about these changes. The survey included some features to reflect on response reliability. We show that the tool seems fairly reliable for comparative analysis and helped to identify the types of firms and contexts where business coaching support appears more effective.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-11-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85933611","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Don’t you forget about me: Independence of evaluations from the perspective of US evaluation clients—An exploratory study","authors":"F. Schmidli, Lyn E. Pleger, Susanne Hadorn","doi":"10.1177/13563890221138876","DOIUrl":"https://doi.org/10.1177/13563890221138876","url":null,"abstract":"Policy evaluation literature has stressed the importance of independence of evaluations to guarantee objective evidence collection. The evaluator–client relationship is critical in this respect, since it contains inherent tensions due to the necessity for independent assessments alongside requirements for increased responsiveness to clients’ interests. Despite this distinct relationship, the client perspective has only recently received attention in research. This article presents findings from a survey among US evaluation clients and compares these to existing evidence from Switzerland. Unlike previous studies, we distinguish between constructive and destructive client influences. We show that professional experience and client familiarity with evaluation standards increase the likelihood of constructive influences aimed at improving evaluation results. Nevertheless, the findings indicate that dissatisfaction with an evaluation increases client’s attempts at influence that may be destructive. By discussing both motives behind influence and potential preventive measures, this article seeks to contribute to the increased social impact of policy evaluations.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-11-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76584634","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}