Sebastian Castro-Alvarez,Laura F Bringmann,Jason Back,Siwei Liu
{"title":"The many reliabilities of psychological dynamics: An overview of statistical approaches to estimate the internal consistency reliability of intensive longitudinal data.","authors":"Sebastian Castro-Alvarez,Laura F Bringmann,Jason Back,Siwei Liu","doi":"10.1037/met0000778","DOIUrl":"https://doi.org/10.1037/met0000778","url":null,"abstract":"Reliability is a key concept in psychology that has been broadly studied since the introduction of Cronbach's α, which is a measure of internal consistency. Despite its importance, reliability has been relatively understudied when dealing with intensive longitudinal data. Although intensive longitudinal measurements are often considered more ecologically valid and less prone to recall bias than survey data collected using traditional methods, there is no warranty that they are more reliable. Hence, empirical researchers need tools to study and report the reliability of the scales used in intensive longitudinal research. In recent years, psychologists have proposed different approaches to estimate the reliability of scales and items used when studying psychological dynamics. However, it is unclear how these approaches compare to one another, making it difficult to determine what options researchers have given a particular data set and specific research questions. Specifically, these approaches estimate reliability indices based on different statistical models, such as linear multilevel analysis, vector autoregressive models, and dynamic factor models. Furthermore, while some methods involve estimating one reliability index for the scores that applies to the whole sample, others estimate person-specific reliability indices. This wide variety of approaches can provoke some confusion. In this article, we aim to bridge this gap by reviewing and highlighting the similarities and differences of different methods used to estimate the reliability of intensive longitudinal data. We also showcase their application with empirical data. (PsycInfo Database Record (c) 2025 APA, all rights reserved).","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"352 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-10-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145319245","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Meir S. Barneron, Tamar Kennet-Cohen, Dvir Kleper, Tzur M. Karelitz
{"title":"Predictive validity of selection tools: The critical role of applicant-pool composition.","authors":"Meir S. Barneron, Tamar Kennet-Cohen, Dvir Kleper, Tzur M. Karelitz","doi":"10.1037/met0000795","DOIUrl":"https://doi.org/10.1037/met0000795","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"72 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-10-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145295069","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jesse Gervais, Geneviève Lefebvre, Erica E. M. Moodie
{"title":"Causal mediation analysis with two mediators: A comprehensive guide to estimating total and natural effects across various multiple mediators setups.","authors":"Jesse Gervais, Geneviève Lefebvre, Erica E. M. Moodie","doi":"10.1037/met0000781","DOIUrl":"https://doi.org/10.1037/met0000781","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"123 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-10-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145295074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Supplemental Material for The Many Reliabilities of Psychological Dynamics: An Overview of Statistical Approaches to Estimate the Internal Consistency Reliability of Intensive Longitudinal Data","authors":"","doi":"10.1037/met0000778.supp","DOIUrl":"https://doi.org/10.1037/met0000778.supp","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"11 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-10-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145296318","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Supplemental Material for Predictive Validity of Selection Tools: The Critical Role of Applicant-Pool Composition","authors":"","doi":"10.1037/met0000795.supp","DOIUrl":"https://doi.org/10.1037/met0000795.supp","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"21 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145241978","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The repeated adjustment of measurement protocols method for developing high-validity text classifiers.","authors":"Alex Goddard, Alex Gillespie","doi":"10.1037/met0000787","DOIUrl":"10.1037/met0000787","url":null,"abstract":"<p><p>The development and evaluation of text classifiers in psychology depends on rigorous manual coding. Yet, the evaluation of manual coding and computational algorithms is usually considered separately. This is problematic because developing high-validity classifiers is a repeated process of identifying, explaining, and addressing conceptual and measurement issues during both the manual coding and classifier development stages. To address this problem, we introduce the Repeated Adjustment of Measurement Protocols (RAMP) method for developing high-validity text classifiers in psychology. The RAMP method has three stages: manual coding, classifier development, and integrative evaluation. These stages integrate the best practices of content analysis (manual coding), data science (classifier development), and psychology (integrative evaluation). Central to this integration is the concept of an inference loop, defined as the process of maximizing validity through repeated adjustments to concepts and constructs, guided by push-back from the empirical data. Inference loops operate both within each stage of the method and across related studies. We illustrate RAMP through a case study, where we manually coded 21,815 sentences for misunderstanding (Krippendorff's α = .79), and developed a rule-based classifier (Matthews correlation coefficient [MCC] = 0.22), a supervised machine learning classifier (Bidirectional Encoder Representations From Transformers; MCC = 0.69) and a large language model classifier (GPT-4o; MCC = 0.47). By integrating manual coding and classifier development stages, we were able to identify and address a concept validity problem with misunderstandings. RAMP advances existing methods by operationalizing validity as an ongoing dynamic process, where concepts and constructs are repeatedly adjusted toward increasingly widespread intersubjective agreement on their utility. (PsycInfo Database Record (c) 2025 APA, all rights reserved).</p>","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.8,"publicationDate":"2025-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145233362","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Supplemental Material for Causal Mediation Analysis With Two Mediators: A Comprehensive Guide to Estimating Total and Natural Effects Across Various Multiple Mediators Setups","authors":"","doi":"10.1037/met0000781.supp","DOIUrl":"https://doi.org/10.1037/met0000781.supp","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"55 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145254622","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ivan Jacob Agaloos Pesigan, Michael A Russell, Sy-Miin Chow
{"title":"Inferences and effect sizes for direct, indirect, and total effects in continuous-time mediation models.","authors":"Ivan Jacob Agaloos Pesigan, Michael A Russell, Sy-Miin Chow","doi":"10.1037/met0000779","DOIUrl":"10.1037/met0000779","url":null,"abstract":"<p><p>Mediation modeling using longitudinal data is an exciting field that captures the interrelations in dynamic changes, such as mediated changes, over time. Even though discrete-time vector autoregressive approaches are commonly used to estimate indirect effects in longitudinal data, they have known limitations due to the dependency of inferential results on the time intervals between successive occasions and the assumption of regular spacing between measurements. Continuous-time vector autoregressive models have been proposed as an alternative to address these issues. Previous work in the area (e.g., Deboeck & Preacher, 2015; Ryan & Hamaker, 2021) has shown how the direct, indirect, and total effects, for a range of time-interval values, can be calculated using parameters estimated from continuous-time vector autoregressive models for causal inferential purposes. However, both standardized effects size measures and methods for calculating the uncertainty around the direct, indirect, and total effects in continuous-time mediation have yet to be explored. Drawing from the mediation model literature, we present and compare results using the delta, Monte Carlo, and parametric bootstrap methods to calculate SEs and confidence intervals for the direct, indirect, and total effects in continuous-time mediation for inferential purposes. Options to automate these inferential procedures and facilitate interpretations are available in the cTMed R package. (PsycInfo Database Record (c) 2025 APA, all rights reserved).</p>","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":" ","pages":""},"PeriodicalIF":7.8,"publicationDate":"2025-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12494154/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145213089","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Supplemental Material for The Repeated Adjustment of Measurement Protocols Method for Developing High-Validity Text Classifiers","authors":"","doi":"10.1037/met0000787.supp","DOIUrl":"https://doi.org/10.1037/met0000787.supp","url":null,"abstract":"","PeriodicalId":20782,"journal":{"name":"Psychological methods","volume":"29 1","pages":""},"PeriodicalIF":7.0,"publicationDate":"2025-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145254633","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}