Evaluation Quality Assessment Frameworks: A Comparative Assessment of Their Strengths and Weaknesses

IF 1.1 3区社会学 Q2 SOCIAL SCIENCES, INTERDISCIPLINARY

American Journal of Evaluation Pub Date : 2022-12-20 DOI:10.1177/10982140211062815

A. Pollard, K. Forss

{"title":"Evaluation Quality Assessment Frameworks: A Comparative Assessment of Their Strengths and Weaknesses","authors":"A. Pollard, K. Forss","doi":"10.1177/10982140211062815","DOIUrl":null,"url":null,"abstract":"Quality Assessment Frameworks (QAFs) are standardized templates that include pre-defined criteria, open questions, or a combination to assess the quality of evaluations. They support organizations to limit risks associated with poor-quality evaluation. Qualitative research focused on four QAFs shows that five dimensions of quality are assessed within the frameworks that include pre-defined criteria. These are aligned with the criteria that audiences of evaluation identify as important to assessing quality: substantive findings, robust and appropriate methodology, accessibility, inclusive processes and analysis of wider systems. Consequently, these QAFs are likely to support critical assessment of the main dimensions of quality that are relevant to audiences of evaluation. Audiences also note synergies between these five dimensions of quality and appropriate contextualization as crucial to assessing quality. Consequently, the structure and categorization of quality within QAFs which only include pre-defined criteria risks limiting reviewers’ ability to reflect on these synergies and contextualize assessments.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"190 - 210"},"PeriodicalIF":1.1000,"publicationDate":"2022-12-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Journal of Evaluation","FirstCategoryId":"90","ListUrlMain":"https://doi.org/10.1177/10982140211062815","RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"SOCIAL SCIENCES, INTERDISCIPLINARY","Score":null,"Total":0}

引用次数: 0

Abstract

Quality Assessment Frameworks (QAFs) are standardized templates that include pre-defined criteria, open questions, or a combination to assess the quality of evaluations. They support organizations to limit risks associated with poor-quality evaluation. Qualitative research focused on four QAFs shows that five dimensions of quality are assessed within the frameworks that include pre-defined criteria. These are aligned with the criteria that audiences of evaluation identify as important to assessing quality: substantive findings, robust and appropriate methodology, accessibility, inclusive processes and analysis of wider systems. Consequently, these QAFs are likely to support critical assessment of the main dimensions of quality that are relevant to audiences of evaluation. Audiences also note synergies between these five dimensions of quality and appropriate contextualization as crucial to assessing quality. Consequently, the structure and categorization of quality within QAFs which only include pre-defined criteria risks limiting reviewers’ ability to reflect on these synergies and contextualize assessments.

查看原文本刊更多论文

评估质量评估框架：优缺点的比较评估

质量评估框架（QAF）是标准化的模板，包括预定义的标准、开放式问题或评估评估质量的组合。他们支持各组织限制与低质量评估相关的风险。以四个质量保证框架为重点的定性研究表明，在包括预定义标准的框架内评估了质量的五个维度。这些都符合评价对象认为对评估质量很重要的标准：实质性调查结果、稳健和适当的方法、可及性、包容性流程和对更广泛系统的分析。因此，这些质量保证框架可能支持对与评估对象相关的主要质量维度进行批判性评估。观众还注意到，质量的这五个维度与适当的情境化之间的协同作用对评估质量至关重要。因此，仅包括预定义标准的质量保证框架内的质量结构和分类有可能限制审查人员反思这些协同作用和将评估置于情境中的能力。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

American Journal of Evaluation SOCIAL SCIENCES, INTERDISCIPLINARY-

CiteScore

4.40

自引率

11.80%

发文量

期刊介绍： The American Journal of Evaluation (AJE) publishes original papers about the methods, theory, practice, and findings of evaluation. The general goal of AJE is to present the best work in and about evaluation, in order to improve the knowledge base and practice of its readers. Because the field of evaluation is diverse, with different intellectual traditions, approaches to practice, and domains of application, the papers published in AJE will reflect this diversity. Nevertheless, preference is given to papers that are likely to be of interest to a wide range of evaluators and that are written to be accessible to most readers.