{"title":"ChatGPT for complex text evaluation tasks","authors":"Mike Thelwall","doi":"10.1002/asi.24966","DOIUrl":"https://doi.org/10.1002/asi.24966","url":null,"abstract":"<p>ChatGPT and other large language models (LLMs) have been successful at natural and computer language processing tasks with varying degrees of complexity. This brief communication summarizes the lessons learned from a series of investigations into its use for the complex text analysis task of research quality evaluation. In summary, ChatGPT is very good at understanding and carrying out complex text processing tasks in the sense of producing plausible responses with minimum input from the researcher. Nevertheless, its outputs require systematic testing to assess their value because they can be misleading. In contrast to simple tasks, the outputs from complex tasks are highly varied and better results can be obtained by repeating the prompts multiple times in different sessions and averaging the ChatGPT outputs. Varying ChatGPT's configuration parameters from their defaults does not seem to be useful, except for the length of the output requested.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 4","pages":"645-648"},"PeriodicalIF":2.8,"publicationDate":"2024-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.24966","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143622691","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Associating cognitive abilities with naturalistic search behavior","authors":"Tung Vuong, Pritom Kumar Das, Tuukka Ruotsalo","doi":"10.1002/asi.24963","DOIUrl":"https://doi.org/10.1002/asi.24963","url":null,"abstract":"<p>Differences in cognitive abilities affect search behaviors, but this has mostly been observed in laboratory experiments. There is limited research on how users search for information in real-world, naturalistic settings and how real-world search behaviors relate to cognitive abilities. In this study, we investigated a wide range of behavioral data captured from real-life search tasks, their association with users' cognitive abilities, and the potential for automatically inferring cognitive abilities from these data. Furthermore, we aimed to determine the data quantity and monitoring duration needed to effectively estimate cognitive abilities from naturalistic behavior. Twenty individuals with βvarying cognitive abilities participated in the experiments in which their everyday search behavior was continuously recorded for 14 days. Their cognitive ability was evaluated through standard tests conducted individually. Data consisted of over 800 h of monitoring, including 2022 queries extracted from 1,442,447 screen frames and associated operating system logs. Using these data, naturalistic search behaviors were associated with cognitive abilities, and predictive models were trained. The results showed that lower selective attention was found to be associated with longer dwelling on selected search results. Faster psychomotor speed and higher fluid intelligence were found to be associated with a greater amount of text read on selected pages. Predictive models exhibited small error rates in predicting cognitive abilities.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 4","pages":"665-685"},"PeriodicalIF":2.8,"publicationDate":"2024-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.24963","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143622444","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Anthony J. Million, Jeremy York, Sara Lafia, Libby Hemphill
{"title":"Data, not documents: Moving beyond theories of information-seeking behavior to advance data discovery","authors":"Anthony J. Million, Jeremy York, Sara Lafia, Libby Hemphill","doi":"10.1002/asi.24962","DOIUrl":"https://doi.org/10.1002/asi.24962","url":null,"abstract":"<p>Many theories of human information behavior (HIB) assume that information objects are in text document format. This paper argues four important HIB theories are insufficient for describing users' search strategies for data because of assumptions about the attributes of objects that users seek. We first review and compare four HIB theories: Bates' <i>berrypicking</i>, Marchionni's <i>electronic information search</i>, Dervin's <i>sense-making</i>, and Meho and Tibbo's <i>social scientist information-seeking</i>. All four theories assume that information-seekers search for text documents. Next, we compare these theories to search behavior by analyzing Google Analytics data from the Inter-university Consortium for Political and Social Research (ICPSR). Users took direct, scenic, and orienting paths when searching for data. We also interviewed ICPSR users (<i>n</i> = 20), and they said they needed dataset documentation and contextual information to find data. However, Dervin's <i>sense-making</i> alone cannot explain the information-seeking behaviors that we observed. Instead, what mattered most were object attributes determined by the type of information that users sought (i.e., data, not documents). We conclude by suggesting an alternative frame for building user-centered data discovery tools.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 4","pages":"649-664"},"PeriodicalIF":2.8,"publicationDate":"2024-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.24962","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143622443","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Katrina Fenlon, Peter Organisciak, Andrea Thomer, Nicholas M. Weber
{"title":"Conceptual models of the sociotechnical: Introduction to special issue","authors":"Katrina Fenlon, Peter Organisciak, Andrea Thomer, Nicholas M. Weber","doi":"10.1002/asi.24958","DOIUrl":"https://doi.org/10.1002/asi.24958","url":null,"abstract":"<p>This special issue of the “Journal of the Association for Information Science and Technology” examines conceptual models as products of, and tools for, critical inquiry in Information Science (IS). The papers included in this issue present diverse perspectives on how conceptual models impact sociotechnical systems, spanning topics such as knowledge organization, representation, and information system design. Key themes include the intersection of model development with ethical considerations, the historical and future implications of conceptual modeling decisions, and the potential for conceptual models to address issues of power, representation, and justice in emerging technologies. This introduction situates the contributions within broader discussions of conceptual modeling in IS and highlights the field's unique approach to reflexive critique and sociotechnical analysis.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 2","pages":"349-352"},"PeriodicalIF":2.8,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143111890","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Integration patterns in the use of metadata for data sense-making during relevance evaluation: An interpretable deep learning-based prediction","authors":"Qiao Li, Ping Wang, Chunfeng Liu, Xueyi Li, Jingrui Hou","doi":"10.1002/asi.24961","DOIUrl":"https://doi.org/10.1002/asi.24961","url":null,"abstract":"<p>Integrating diverse cues from metadata to make sense of retrieved data during relevance evaluation is a crucial yet challenging task for data searchers. However, this integrative task remains underexplored, impeding the development of effective strategies to address metadata's shortcomings in supporting this task. To address this issue, this study proposes the “Integrative Use of Metadata for Data Sense-Making” (IUM-DSM) model. This model provides an initial framework for understanding the integrative tasks performed by data searchers, focusing on their integration patterns and associated challenges. Experimental data were analyzed using an interpretable deep learning-based prediction approach to validate this model. The findings offer preliminary support for the model, revealing that data searchers engage in integrative tasks to utilize metadata effectively for data sense-making during relevance evaluation. They construct coherent mental representations of retrieved data by integrating systematic and heuristic cues from metadata through two distinct patterns: within-category integration and across-category integration. This study identifies key challenges: within-category integration entails comparing, classifying, and connecting systematic or heuristic cues, while across-category integration necessitates considerable effort to integrate cues from both categories. To support these integrative tasks, this study proposes strategies for mitigating these challenges by optimizing metadata layouts and developing intelligent data retrieval systems.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 3","pages":"621-641"},"PeriodicalIF":2.8,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.24961","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143424141","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The role of online search platforms in scientific diffusion","authors":"Kyriakos Drivas","doi":"10.1002/asi.24959","DOIUrl":"https://doi.org/10.1002/asi.24959","url":null,"abstract":"<p>After the launch of Google Scholar older papers experienced an increase in their citations, a finding consistent with a reduction in search costs and introduction of ranking algorithms. I employ this observation to examine how recombination of science takes place in the era of online search platforms. The findings show that as papers become more discoverable, their knowledge is diffused beyond their own broad field. Results are mixed when examining knowledge diffusion within the same field. The results contribute to the ongoing debate of narrowing of science. While there might a general reduction in recombination of knowledge across distant fields over the last decades, online search platforms are not the culprits.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 3","pages":"580-603"},"PeriodicalIF":2.8,"publicationDate":"2024-10-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.24959","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143424313","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Falling behind again? Characterizing and assessing older adults' algorithm literacy in interactions with video recommendations","authors":"Yuhao Zhang, Jiqun Liu","doi":"10.1002/asi.24960","DOIUrl":"https://doi.org/10.1002/asi.24960","url":null,"abstract":"<p>Algorithms play a significant role in shaping our experiences of interacting with intelligent information systems but also inherit and amplify data biases, potentially leading to unfair decisions or discriminatory outcomes. This motivates us to investigate users' <i>algorithm literacy</i>, which covers the awareness and knowledge of algorithms and the skills to intervene in the operations of personalization algorithms when interacting with recommendation systems. Since vulnerable groups are more likely to suffer from the negative consequences of algorithmic decision-making, investigating algorithm literacy among such groups is critical. This study aims to examine older adults' algorithm literacy, who are often considered a vulnerable group and labeled as digital laggards in contemporary information society. The empirical evidence collected from 21 participants in in-depth interviews and cognitive mapping studies demonstrated that almost all participants are algorithm-aware to some extent and identified (1) three types of information and sources collected by algorithms in user understanding, (2) two paradigms of how respondents understand personalized recommendations, and (3) two sets of strategies they develop to employ algorithms for improving user experience. The findings shed light on designing human-centered intelligent information systems for unbiased personalization and developing a more inclusive AI-assisted society that equally benefits people of all ages.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 3","pages":"604-620"},"PeriodicalIF":2.8,"publicationDate":"2024-10-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143424248","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Exploring the digital gray zone of online medicinal markets emerging from search","authors":"Kristofer Rolf Söderström, Olof Sundin","doi":"10.1002/asi.24956","DOIUrl":"https://doi.org/10.1002/asi.24956","url":null,"abstract":"<p>This explorative study investigates the emergence of gray zone markets from search engines amidst the global expansion of online markets. With the analytical approach of infrastructural inversion, we examine how the search infrastructure constructs access to a gray zone market including both authorized online pharmacies and unauthorized vendors. Using Sweden and Google Search as a case, we explore the online presence of three products (vitamin D, paracetamol, and Viagra), through search engine result page analysis, web crawling, and network analysis. Infrastructural inversion unveils the typically invisible mechanisms of search engines, considering user queries, algorithmic priorities, SEO practices, and pharmacy regulations. We find gray zones only emerge in searches for erectile disfunction medicinal products and information, where unauthorized vendors successfully competed for visibility in search engine rankings. A complex web of conditions can steer consumers toward gray zone markets, complicating the access to safe and regulated medicinal products. This can expose individuals to risks associated with unverified medicinal products, but also challenges the integrity of the online health information infrastructure.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"75 13","pages":"1498-1514"},"PeriodicalIF":2.8,"publicationDate":"2024-10-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.24956","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142860870","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"When data sharing is an answer and when (often) it is not: Acknowledging data-driven, non-data, and data-decentered cultures","authors":"Isto Huvila, Luanne S. Sinnamon","doi":"10.1002/asi.24957","DOIUrl":"https://doi.org/10.1002/asi.24957","url":null,"abstract":"<p>Contemporary research and innovation policies and advocates of data-intensive research paradigms continue to urge increased sharing of research data. Such paradigms are underpinned by a pro-data, normative data culture that has become dominant in the contemporary discourse. Earlier research on research data sharing has directed little attention to its alternatives as more than a deficit. The present study aims to provide insights into researchers' perspectives, rationales and practices of (non-)sharing of research data in relation to their research practices. We address two research questions, (RQ1) what underpinning patterns can be identified in researchers' (non-)sharing of research data, and (RQ2) how are attitudes and data-sharing linked to researchers' general practices of conducting their research. We identify and describe <i>data-decentered culture</i> and <i>non-data culture</i> as alternatives and parallels to the <i>data-driven culture</i>, and describe researchers de-inscriptions of how they resist and appropriate predominant notions of data in their data practices by problematizing the notion of data, asserting exceptions to the general case of data sharing, and resisting or opting out from data sharing.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"75 13","pages":"1515-1530"},"PeriodicalIF":2.8,"publicationDate":"2024-10-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/asi.24957","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142861285","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Negative consequences of information gatekeeping through algorithmic technologies: An Annual Review of Information Science and Technology (ARIST) paper","authors":"Devendra Potnis, Iman Tahamtan, Luke McDonald","doi":"10.1002/asi.24955","DOIUrl":"https://doi.org/10.1002/asi.24955","url":null,"abstract":"<p>Rarely any study investigates <i>how</i> information gatekeeping through the solutions and services enabled by algorithms, hereafter referred to as algorithmic technologies (AT), creates negative consequences for the users. To fill this gap, this state-of-the-art review analyzes 229 relevant articles from diverse academic disciplines. We employed thematic analysis to identify, analyze, classify, and reveal the chain reactions among the negative consequences. We found that the gatekeeping of information (text, audio, video, and graphics) through AT like artificial intelligence (e.g., chatbots, large language models, machine learning, robots), decision support systems (used by banks, grocery stores, police, etc.), hashtags, online gaming platforms, search technologies (e.g., voice assistants, ChatGPT), and Web 3.0 (e.g., Internet of Things, non-fungible tokens) creates or reinforces cognitive vulnerability, economic divide and financial vulnerability, information divide, physical vulnerability, psychological vulnerability, and social divide virtually and in the offline world. Theoretical implications include the hierarchical depiction of the chain reactions among the primary, secondary, and tertiary divides and vulnerabilities. To mitigate these negative consequences, we call for concerted efforts using top-down strategies for governments, organizations, and technology experts to attain more transparency, accountability, ethical behavior, and moral practices, and bottom-up strategies for users to be more alert, discerning, critical, and proactive.</p>","PeriodicalId":48810,"journal":{"name":"Journal of the Association for Information Science and Technology","volume":"76 1","pages":"262-288"},"PeriodicalIF":2.8,"publicationDate":"2024-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143111437","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}