Realist evaluation and its role in the stages of explanatory research based on critical realism

Abstract

ABSTRACTThis article advocates for the validity of Realist Evaluation (RE) as a manifestation of Critical Realism in evaluation research despite criticisms suggesting that the former disregards principles from Bhaskarian ontology. Specifically, I argue that critics overstate RE's philosophical actualism in their argument that its inclination towards technocratic knowledge impedes its scrutiny of stratified social systems. Notwithstanding its limitations in fully elucidating causal structural mechanisms in social inquiry, I argue that RE's research rationale can contribute to the stages of explanatory research based on CR by bridging the comprehension of mechanisms linked to micro-interventions with exploring potential structural forces operating across levels of reality. To illustrate this point, I present a case of a policy evaluation study I led that demonstrates how a RE-oriented research design facilitated a transition from investigating specific programme mechanisms within the interventions' context to proposing transcendental questions about the governance of the education system in Colombia.KEYWORDS: Realist evaluationstratificationactualismstructure and agencycausality AcknowledgementsMany of the ideas in the article emerged from valuable discussions during the IACR 2022 annual critical realism conference held in The Hague, The Netherlands. I was fortunate to have Margaret Archer, the influential and inspiring CR scholar who sadly died this year, as part of the audience during my presentation of a preliminary version of this paper.Disclosure statementNo potential conflict of interest was reported by the author(s).Notes1 Roy Bhaskar is considered a founder of this philosophical tradition (Maisuria and Banfield Citation2022).2 Retroduction, as a mode of reasoning, is central to CR. I define this concept and will refer to this argument later in the article.3 Indeed, in his Realist Manifesto, Pawson (Citation2013) presents Bhaskar’s Realist Theory of Science—published in the late 1970s—as one of the foremost intellectual precursors of RE.4 Consequently, RE scholars maintain that mainstream approaches to programme assessment, like impact evaluation (highly deductive) or constructivist case studies (highly inductive), do not offer sufficient guidance for comprehensive scrutiny of programme mechanisms (Tikly Citation2015).5 As RE scholars explain, these hypotheses may arise through the examination of various sources, such as pertinent empirical or academic literature addressing the phenomena of interest. Additionally, discussions with specific stakeholders, such as programme designers or implementers, can help distil potential explanations for why particular interventions may be effective in specific contexts.6 That is, practising a social science that ‘necessarily takes the form of explanatory critique and so directly impinges on the project of human emancipation from reproduced structures of domination that constrain our essential freedom’ (Bhaskar and Hartwig Citation2010, 75)7 However, this is unsurprising as many of the critiques raised by Alderson to RE in her book backed up in Porter’s papers where the latter problematises Pawson’s rethoric vis-à-vis evaluation.8 Interestingly, even Porter (Citation2015b) argues that ‘[t]he differences between critical realism and realist evaluation are not as significant as Pawson contends’ (2015b, 65).9 Indeed, this argument about different levels of analysis aligns with Pawson’s responses to Potter’s critique of RE: ‘Thus, for instance, if I was investigating the mixed fortunes of traffic calming schemes, I wouldn’t expend too much effort on capitalist modes of production (…) If, by contrast, the inquiry was about how forms of late western capitalism are giving way to Chinese state capitalism, I might well examine the former’s self-imploding, too-big-to-fail institutions and their material capacity to defy regulation’ (Pawson Citation2016b, 54).10 In Bhaskars (Citation2016) words, ‘[r]etroduction involves imagining a model of a mechanism that, if it were real, would account for the phenomenon in question’ (79).11 For instance, in her book Alderson (Citation2021) makes the following assertion as she analysis the role of ontology in medical research: ‘Critical realists are very interested in observing and analysing how individuals’ beliefs change (…), but they do not try to direct individuals as RE does. Instead, CR concepts of health promotion include changing structures to advance justice and human flourishing generally’ (34-35).12 Indeed, this argument aligns with Pawson’s responses to Porter’s critique, as the former emphasizes the significance of considering different levels of analysis when examining relevant causal mechanisms.13 Notably, this notion of layered theories echoes the conception of laminated systems coined earlier by Bhaskar and Danermark (Citation2006) to conceptualize how causal explanations deem the consideration ‘of a multiplicity of mechanisms, potentially of radically different kinds (and potentially demarcating the site of distinct disciplines) corresponding to different levels or aspects of reality’ (288).14 Bhaskar (Citation2016) defines abduction as mode of reasoning that ‘involves redescription or recontextualisation, most usually (in critical realist research) in terms of a causal mechanism or process that serves to explain the state, condition or happening referred to’ (79).15 As Sorinola et al. (Citation2017) further clarify, in RE ‘Context consists of the broader historical, cultural, economic, geographical, and structural factors that exist at the time of the initiative which includes individuals (the characteristics and capacities of the stakeholders); interpersonal (relationships); institutional settings (rules, norms and customs) and infrastructural system (the wider social, economic and cultural setting)’ (423).16 To this point, it is worth mentioning that part of the critique of RE by Alderson (Citation2021) highlights its supposed lack of commitment to critical theory, therefore failing ‘to promote justice and equality’ (34). However, that notion of a commitment, a priori, to specific values seems to contradict Bhaskar himself. Consider, for instance, the following quote: ‘For critical realism explanatory theory implies, rather than (as in Horkheimer and Habermas) presupposes, a commitment to emancipation. Thus we need not preface our search for explanatory mechanisms with our interest in emancipation; on the contrary, our interest in emancipation can flow from the search’ (Bhaskar Citation2016, 101). Therefore, the commitment to endorse a stratified ontology, more than an explicit declaration of values, grants RE an inherent potential to contribute to emancipatory-oriented research.17 JU stands for Jornada Única.18 As Manzano (Citation2016) explains, ‘[t]he questions asked in these first set of interviews will be mainly exploratory and the wording of those questions should try to ascertain how the programme works for whom and in what circumstances. Questions looking to explore context may ask for interviewees’ experiences of before/during/after the programme was implemented. For example: ‘How was your work different before the programme was implemented?’, ‘Is this new programme going to work for everyone?’, ‘Could you explain to me the types of people and places where you think it may be more effective?’ (Manzano Citation2016, 354)Additional informationNotes on contributorsJuan David ParraJuan David Parra is an Assistant Professor at the Institute of Education Studies from Universidad del Norte (Colombia). He has participated in several evaluation studies and is a precursor of Realist Evaluation (in education) in Latin America. In 2022, he joined an international consortium led by the University of Notre Dame as a technical advisor of USAID’s Supporting Holistic and Actionable Research in Education (SHARE) initiative to advance education learning priorities in low-middle-income countries. His work appears published in peer-reviewed journals such as Third World Quarterly, British Journal of Sociology of Education, International Journal of Educational Development and International Journal of Qualitative Studies in Education. He also contributed with a book chapter in the recent volume on Systems Thinking in International Education and Development edited by NORRAG – Network for International Policies and Cooperation in Education and Training (Edward Elgar Publishing, 2023). He holds a PhD in Development Studies from the International Institute of Social Studies, Erasmus University Rotterdam.