Performance Evaluation最新文献

{"title":"Evaluation during war: Current realities and future possibilities of Ukrainian monitoring and evaluation","authors":"Nataliya Antoniv, H. I. Kalyta, Dmytro Kondratenko, O. Krasovska, I. Kravchuk, I. Lupashko, Liubov Margolina, Larysa Pylgun, Antonina Rishko-Porcescu, Mykhailo Savva, Yulia Zinovieva","doi":"10.1177/13563890221138739","DOIUrl":"https://doi.org/10.1177/13563890221138739","url":null,"abstract":"As of mid-September 2022, there are nearly 17.7 million people in need in Ukraine. As many as 14 million Ukrainians have been forced to flee their homes as a result of Russia’s full-scale invasion of Ukraine, including 7.3 million refugees and 7 million displaced persons within Ukraine. The number of confirmed civilian casualties amounted to 13,212 people.1 18.9 per cent of Ukraine’s landmass is occupied by Russian forces, the frontline is constantly changing and all regions of the country, without exception, are a target of constant missile strikes. Russia is clearly not limiting its firing upon military targets but on medical and educational facilities, residential areas, as well as evacuation and volunteer centres. In such conditions, Ukrainian non-governmental organisations (NGOs), government, businesses and volunteers are working on aiding those in need and restoring the devastated territories. The unfolding conflict has made most civil society organisations (CSOs) focus on humanitarian aid and reorient their activities from development into the humanitarian sphere, where they have had to provide vital supplies and essentials in the shortest possible time to a larger number of civilians, the military or participants of the Territorial Defence Forces.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-11-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"72569283","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Why is impact measurement abandoned in practice? Evidence use in evaluation and contracting for five European Social Impact Bonds","authors":"Debra Hevenstone, Alec Fraser, L. Hobi, Gemma G. M. Geuke","doi":"10.1177/13563890221136890","DOIUrl":"https://doi.org/10.1177/13563890221136890","url":null,"abstract":"Despite broad consensus on the importance of measuring “impact,” the term is not always understood as estimating counterfactual and causal estimates. We examine a type of public sector financing, “Social Impact Bonds,” a scheme where investors front money for public services, with repayment conditional on impact. We examine five cases in four European countries of Social Impact Bonds financing active labor market programs, testing the claim that Social Impact Bonds would move counterfactual causal impact evaluation to the heart of policy. We examine first how evidence was integrated in contracts, second the overall evidence generated and third, given that neither contracts nor evaluations used counterfactual definitions of impact, we explore stakeholders’ perspectives to better understand the reasons why. We find that although most stakeholders wanted the Social Impact Bonds to generate impact estimates, beliefs about public service reform, incentives, and the logic of experimentation led to the acceptance of non-causal definitions.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86450688","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Pivoting to online: The benefits, challenges and possibilities for international programme evaluations","authors":"L. Henley, Nicky Stanley-Clarke, Anuradha Acharya, Smriti Khadka, A. Olsson","doi":"10.1177/13563890221137610","DOIUrl":"https://doi.org/10.1177/13563890221137610","url":null,"abstract":"Global South non-government organizations rely on international funding and aid for continued service delivery. Service evaluation plays a significant role in ensuring compliance and ongoing service funding. Traditional service evaluation approaches could not take place during 2021 due to COVID-19, alternate mechanisms needed to be embraced. This article reports on the benefits and challenges of undertaking service evaluations online during the pandemic and the learnings and possibilities for a post-pandemic world. It emphasizes the importance of translating a relational approach to service evaluation to the online environment. Key learnings include that while some of the benefits of context and in-person connection are lost, taking a relational approach involving careful planning and reflexivity means the online evaluation process can be successful. Undertaking service evaluations online offers possibilities in a post-pandemic world as cost-effective alternatives to the expensive and time-consuming reality of in-person service evaluation across international borders and within development contexts.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86403413","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Ethics of evaluation for socio-ecological transformation: Case-based critical systems analysis of motivation, power, expertise, and legitimacy","authors":"Emily F. Gates, Glenn Page, J. M. Crespo, Mauricio Nuñez Oporto, Juliana Bohórquez","doi":"10.1177/13563890221129640","DOIUrl":"https://doi.org/10.1177/13563890221129640","url":null,"abstract":"Evaluation that supports social, ecological, and governance systems change and transformation raises ethical questions about what and whose worldviews do and should ground evaluative processes. This article illustrates one approach to ethical analysis within a case study of the first phase of an initiative to co-create a monitoring, evaluation, and learning system. The case drew on the principles of Blue Marble Evaluation in partnership with local staff and Indigenous leaders of the Amazon Sacred Headwaters Initiative. The approach uses critical and relational systems thinking to examine the sources of motivation, power, knowledge, and legitimacy that influence and should influence an evaluation system. Results reframe typical early phase evaluation process work from a contractual agreement to a co-created ethical space that engenders the legitimacy of the evaluation process. Contributions include a conceptual framework and process for ethical analysis that could be adapted by others.","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"81492020","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Parameter estimation of Markov modulated fluid arrival processes","authors":"Salah Al-Deen Almousa ,&nbsp;Gábor Horváth ,&nbsp;Miklós Telek","doi":"10.1016/j.peva.2022.102316","DOIUrl":"10.1016/j.peva.2022.102316","url":null,"abstract":"<div><p>Markov modulated discrete arrival processes have a wide literature, including parameter estimation methods based on expectation–maximization (EM). In this paper, we investigate the adaptation of these EM based methods to Markov modulated fluid arrival processes (MMFAP), and conclude that only the generator matrix of the modulating Markov chain of MMFAPs can be approximated by EM based method. For the rest of the parameters, the fluid rates and the fluid variances, we investigate the efficiency of numerical likelihood maximization.</p><p>To reduce the computational complexity of the likelihood computation, we accelerate the numerical inverse Laplace transformation step of the procedure with function fitting.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0166531622000256/pdfft?md5=aedc437af6444b2801e92fd5922ca3f7&pid=1-s2.0-S0166531622000256-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"74301314","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"I/O performance analysis of machine learning workloads on leadership scale supercomputer","authors":"Ahmad Maroof Karimi ,&nbsp;Arnab K. Paul ,&nbsp;Feiyi Wang","doi":"10.1016/j.peva.2022.102318","DOIUrl":"10.1016/j.peva.2022.102318","url":null,"abstract":"<div><p>The popularity of machine learning<span> technologies and frameworks has led to an increasingly large number of machine learning workloads running on high-performance computing (HPC) clusters. The ML workflows are readily being adopted in diverse computational fields such as Biology, Physics, Materials, and Computer Science. The I/O behavior of the emerging ML workloads distinctly differs from the traditional HPC workloads, such as simulation or checkpoint/restart-based HPC I/O behavior. Additionally, the ML workloads have also pushed for the utilization of GPUs or a combination of CPUs and GPUs in addition to using only CPUs for computational tasks. The diverse and complex I/O behavior of ML workloads requires extensive study and is critical for the efficient performance of various layers of the I/O stack and the overall performance of HPC workloads. This work aims to fill the gap in understanding the I/O behavior of emerging ML workloads by providing an in-depth analysis of ML jobs running on large-scale leadership HPC systems. In particular, we have analyzed the behavior of jobs based on the scale of the jobs, the science domains, and the processing units used by the ML jobs. The analysis was performed on 23,000 ML jobs collected from one year of Darshan logs running on Summit, which is one of the fastest supercomputers<span>. We also collect the CPU and GPU usage of 15,165 ML jobs by merging the Darshan dataset with the power usage of the processing units on Summit. Therefore, this paper is able to provide a systematic I/O characterization of ML workloads on a leadership scale HPC machine to understand how the I/O behavior differs for workloads across various science domains, the scale of workloads, and processing units and analyze the usage of parallel file system and burst buffer by ML I/O workloads. We have made several observations regarding I/O performances and access patterns through various analytical studies and discuss the important lessons learnt from the perspective of a ML user and a storage architect for emerging ML workloads running on large-scale supercomputers.</span></span></p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"79762898","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Low-complexity scheduling algorithms with constant queue length and throughput guarantees","authors":"Subrahmanya Swamy Peruru ,&nbsp;Aravind Srinivasan ,&nbsp;Radha Krishna Ganti ,&nbsp;Krishna Jagannathan","doi":"10.1016/j.peva.2022.102310","DOIUrl":"10.1016/j.peva.2022.102310","url":null,"abstract":"<div><p><span>Distributed scheduling algorithms based on carrier sense multiple access (CSMA) are optimal in terms of the throughput and the steady-state queue lengths. However, they take a prohibitively long time to reach the steady-state, often exponential in the network size. Therefore for large networks that operate over a finite time horizon, apart from the guarantees on the steady-state queue lengths, performance guarantees on the short-term (</span><em>i.e.</em>, transient) queuing behaviour are also required. To that end, we propose distributed scheduling algorithms that are guaranteed to have <span><math><mrow><mi>O</mi><mrow><mo>(</mo><mn>1</mn><mo>)</mo></mrow></mrow></math></span> expected queue lengths not just in the steady-state but at every time instant, where <span><math><mrow><mi>O</mi><mrow><mo>(</mo><mi>⋅</mi><mo>)</mo></mrow></mrow></math></span> is with respect to the network size. Further, our algorithms have <span><math><mrow><mi>O</mi><mrow><mo>(</mo><mn>1</mn><mo>)</mo></mrow></mrow></math></span><span> complexity and support a constant fraction of the maximum throughput for typical wireless topologies. The central idea of our algorithms is to resolve collisions among pairs of conflicting nodes by assigning a master–follower hierarchy. The master–follower hierarchy can either be chosen randomly or based on the topology of the conflict graph, leading to different performance guarantees.</span></p><p><span>In addition to these hierarchical collision resolution algorithms, which are primarily designed for the conflict graph-based interference model, we also propose an Aloha-based algorithm for the </span><span><math><mi>K</mi></math></span><em>-neighbour collision tolerance</em> interference model, which is a generalization of the conflict graph model. We show that the proposed Aloha-based algorithm supports a constant fraction of the maximum throughput for typical wireless topologies.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76504274","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Improving Massive Access to IoT Gateways","authors":"Erol Gelenbe ,&nbsp;Mert Nakıp,&nbsp;Tadeusz Czachórski","doi":"10.1016/j.peva.2022.102308","DOIUrl":"10.1016/j.peva.2022.102308","url":null,"abstract":"<div><p>IoT networks handle incoming packets from large numbers of IoT Devices (IoTDs) to IoT Gateways. This can lead to the IoT Massive Access Problem that causes buffer overflow, large end-to-end delays and missed deadlines. This paper analyzes a novel traffic shaping method named the Quasi-Deterministic Traffic Policy (QDTP) that mitigates this problem by shaping the incoming traffic without increasing the end-to-end delay or dropping packets. Using queueing theoretic techniques and extensive data driven simulations with real IoT datasets, the value of QDTP is shown as a means to considerably reduce congestion at the Gateway, and significantly improve the IoT network’s overall performance.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0166531622000219/pdfft?md5=a2c57652ab751313e9ab77ef1b537d4c&pid=1-s2.0-S0166531622000219-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88195224","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Asymptotic optimality of speed-aware JSQ for heterogeneous service systems","authors":"Sanidhay Bhambay,&nbsp;Arpan Mukhopadhyay","doi":"10.1016/j.peva.2022.102320","DOIUrl":"https://doi.org/10.1016/j.peva.2022.102320","url":null,"abstract":"<div><p>The Join-the-Shortest-Queue (JSQ) load-balancing scheme is known to minimise the average delay of jobs in homogeneous systems consisting of identical servers. However, it performs poorly in heterogeneous systems where servers have different processing rates. Finding a delay optimal scheme remains an open problem for heterogeneous systems. In this paper, we consider a speed-aware version of the JSQ scheme for heterogeneous systems and show that it achieves delay optimality in the fluid limit. One of the key issues in establishing this optimality result for heterogeneous systems is to show that the sequence of steady-state distributions indexed by the system size is tight in an appropriately defined space. The usual technique for showing tightness by coupling with a suitably defined dominant system does not work for heterogeneous systems. To prove tightness, we devise a new technique that uses the drift of exponential Lyapunov functions. Using the non-negativity of the drift, we show that the stationary queue length distribution has an exponentially decaying tail — a fact we use to prove tightness. Another technical difficulty arises due to the complexity of the underlying state-space and the separation of two time-scales in the fluid limit. Due to these factors, the fluid-limit turns out to be a function of the invariant distribution of a multi-dimensional Markov chain which is hard to characterise. By using some properties of this invariant distribution and using the monotonicity of the system, we show that the fluid limit is has a unique and globally attractive fixed point.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0166531622000281/pdfft?md5=16a46606029e0e71f6846c950586957f&pid=1-s2.0-S0166531622000281-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"137428776","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Performance evaluation of tape library systems","authors":"Ilias Iliadis,&nbsp;Linus Jordan,&nbsp;Mark Lantz,&nbsp;Slavisa Sarafijanovic","doi":"10.1016/j.peva.2022.102312","DOIUrl":"10.1016/j.peva.2022.102312","url":null,"abstract":"<div><p><span>Magnetic tape provides a cost-effective way to retain the exponentially increasing volumes of data being produced. The low cost per gigabyte and the low energy consumption<span> render tape a preferred option over hard disk drives and flash for infrequently accessed data. Assessing the performance of tape library systems is central to achieving appropriate storage provisioning and dimensioning. Performance is affected by the number and the operational characteristics of the tape drives and the </span></span>robotic arms, and the mount and unmount policies deployed. In this paper, we develop a novel analytical model that accurately captures the principal aspects of tape library operation. Several relevant performance measures including the mean waiting time and the mount/unmount rates are derived. The model provides useful insights into the behavior of the tape libraries and yields results that enable a better understanding of the design tradeoffs. The validity of the model developed is confirmed by demonstrating a good agreement of the predicted performance with that obtained by simulation across various configurations. To mitigate the burden on the robotic mechanism, a scheme of accumulating multiple requests before sending them to the tape library is proposed and studied.</p></div>","PeriodicalId":19964,"journal":{"name":"Performance Evaluation","volume":null,"pages":null},"PeriodicalIF":2.2,"publicationDate":"2022-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90450259","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}