Gaming the Metrics最新文献

{"title":"Fake Scientists on Editorial Boards Can Significantly Enhance the Visibility of Junk Journals","authors":"B. Morgenstern","doi":"10.7551/mitpress/11087.003.0019","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0019","url":null,"abstract":"veniently “measure” the performance of scientists and research institutes without actually looking at the research work that is to be “measured.” As Barbour and Stell put it in their contribution to this volume (chapter 11), metrics are “attempts to measure the unmeasurable.” Some absurd consequences of these attempts have been discussed in the previous chapters of this book. Metrics are frequently criticized for being “unfair” or for “distorting” the thing that is to be measured, and there is much debate about alternative metrics that might be more appropriate than the metrics that are currently in use (Jennifer Lin, this volume, chapter 16). In contrast to this viewpoint, I want to argue here that it is fundamentally impossible to measure research work quantitatively. Comparing things quantitatively to each other assumes that they are, in principle, of the same quality. One can, for example, compare the weights of physical objects to each other, since they have the same quality mass. But one cannot quantitatively compare the weight of one object to, say, the color or the speed of another object, since weight, color, and speed are different qualities. Similarly, one can compare the productivity of workers that are doing, in principle, the same type of work. One can say, for example, that surgeon A carries out ten percent more operations per year, with a given success rate, than does her colleague, surgeon B, under similar conditions. But things are different if it comes to research work— at least if we are talking about basic research. Research is about discoveries, inventions, and thoughts that are, by their nature, novel and different from previous discoveries, inventions, or thoughts. It is thus impossible to compare research results quantitatively to each other. It would be utterly absurd to say that, for example, the discovery of the citric acid cycle is five times more than Ukkonen’s proof that the suffix tree for a string of characters can be calculated in linear time. 15 Fake Scientists on Editorial Boards Can Significantly Enhance the Visibility of Junk Journals","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128784870","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Crack Open the Make Believe: Counterfeit, Publication Ethics, and the Global South","authors":"M. Jacob","doi":"10.7551/mitpress/11087.003.0024","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0024","url":null,"abstract":"concept of “counterfeit.” The research context of my short intervention draws on ethnographic and archival work, engaging the question of how people experience but also imagine legality/illegality. Since 2010, as part of my interest in the category of “publication ethics,” I have been conducting ethnographic observations of the quarterly forum of the global charity Committee on Publication Ethics (COPE). My research also looks at how the category of “research misconduct” has taken form in the context of disciplinary adjudication by regulators (Jacob, 2014, 2016a) and of modern patterns of documentation more generally (Jacob, 2017). In brief, I am as much interested if not more in institutional watchdogs of academic misconduct than I am in alleged perpetrators of academic misconduct. Pausing over the mutually exclusive dichotomy of real versus counterfeit journals, my short intervention approaches the idea of counterfeit by way of making three points in relation to public harm and denunciation, the idea of the authentic, and watchdogs. Through these anchor points, I hope we can better see the eruption of counterfeit scientific journals as more inexorable than strange or outrageous. The idea here is not to justify the counterfeit of academic journals by claiming that counterfeit exists elsewhere; it is also not to exoticize or, worse, romanticize counterfeiters. Rather it is to examine it on its own terms, from the point of view of its craft, and to highlight dexterity as one of its most underexplored aspects. As James Siegel has beautifully shown in his ethnography of counterfeiters in contemporary Indonesia (Siegel, 1998), there exists a certain power in making fake university certificates, or fake divorce certificates, and so on. Aside from being about the financial profit it brings, it is a power for crafting “a sort of authority for one’s self” or “one’s own rubber stamp” and for attesting to one’s creative abilities. Given the transformations of scientific research and publishing over the last thirty years, described extensively in 19 Crack Open the Make Believe: Counterfeit, Publication Ethics, and the Global South","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129337976","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Acknowledgments","authors":"","doi":"10.7551/mitpress/11087.003.0027","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0027","url":null,"abstract":"","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"473 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124678412","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Humor, Hoaxes, and Software in the Search for Academic Misconduct","authors":"A. Lippman","doi":"10.7551/mitpress/11087.003.0026","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0026","url":null,"abstract":"By contrast, a new generation of independent watchdogs and bloggers interested in academic misconduct employ jokes, pranks, witty pseudonyms, and humorous hoaxes as a part of their critique and as tools of investigation. The new watchdogs represent a significant shift from topdown, bureaucratic, institutionalized detection of academic misconduct toward collaborative discussion, detection, and dissemination. Not only is this work often done for free, but it also is often done with and through humor. Until recently, hoaxes within academia targeted authorities— highly regarded scholars, journals, or disciplines. In 1996, New York University physicist Alan Sokal wrote and submitted an article to Social Text. His goal was to test whether a top cultural studies journal in the United States would publish an article rife with nonsensical claims “if (a) it sounded good and (b) it flattered the editors’ ideological preconceptions” (Sokal, 1996). But unlike Sokal’s relatively straightforward hoax, which Social Text accepted and published, the bloggers and pranksters discussed in this chapter reveal shams through elaborate jokes and stings, crowd participation, and the creation of fictional personae. Furthermore, while hoaxes as a genre target reputable institutions and figures of authority, contemporary misconduct watchdogs’ hoaxes and jokes take aim at fraudsters with a sense of humor. I argue that the detection, critique, and mocking of academic gaming and scholarship have taken a carnivalesque turn. I will discuss changes within the focus of critique by comparing two softwarebased, scholarly article generators released a decade apart: the Postmodernism Generator, created in 1996, and SCIgen, created in 2005. While the creator of the Postmodernism Generator playfully mocks renowned humanities scholars’ jargon through producing Dadaist computergenerated papers, the 21 Humor, Hoaxes, and Software in the Search for Academic Misconduct","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133991097","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Retraction Watch: What We’ve Learned and How Metrics Play a Role","authors":"I. Oransky","doi":"10.7551/mitpress/11087.003.0014","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0014","url":null,"abstract":".com) in August 2010 for two reasons: As longtime journalists, we often found that retraction notices were opaque. And sometimes opacity was the best you could hope for; often, notices were misleading or even wrong. We also found that there were great stories behind retractions. We have our own metrics at Retraction Watch, mostly just having to do with traffic to the site each month; we now have, on average, 150,000 unique visitors, and half a million page views. (However, we are not beholden to these metrics, as our revenue does not depend on advertising; we have at various times had generous funding from three foundations, and other income streams including freelance writing fees.) In terms of more traditional metrics, I can say we have been cited in the literature more than a hundred times. That means that if a blog could have an H index, we would have a good one. And it does not hurt when we talk to funders about the impact we are having on publishing practices and transparency. Retraction Watch posts often begin with a tip— mostly a notice of retraction. But we also receive long emails from frustrated researchers, who have been laboring to correct a perceived wrong for months, if not years. We empathize and sympathize with their frustration— it is incredibly hard to get papers retracted from the literature, or even corrected or noted in some way. As an illustration, take a piece by nutrition researcher David Allison and colleagues that appeared in Nature (Allison et al., 2016). They scanned the nutrition literature and found more than two dozen papers that they thought were deeply problematic. And they kept a pretty high bar. You can judge for yourselves, but if you look at the kinds of problems they were looking at in these papers, it was pretty clear something needed to be done. In a few cases, the journals retracted the paper, or published a letter from Allison and his team critiquing the findings, but in many cases the 10 Retraction Watch: What We’ve Learned and How Metrics Play a Role","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130645455","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Interventions: Notes from the Field","authors":"","doi":"10.7551/mitpress/11087.003.0013","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0013","url":null,"abstract":"","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"95 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122533870","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Playing and Being Played by the Research Impact Game","authors":"M. Power","doi":"10.7551/mitpress/11087.003.0005","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0005","url":null,"abstract":"and Keele in the United Kingdom, came equal top in a league table, each with grade point averages of 3.80 (where 4 is a maximum). The Department of Philosophy at Oxford University was ranked tenth with a grade point average (GPA) of 3.40. Ranking systems are widespread, not least in the field of education (see Kehm, this volume, chapter 6). For example, Espeland and Sauder (2007) examine the rankings of US law schools and their effects on the behavior of key organizational participants, such as deans, who are compelled to pay attention to them despite being doubtful of their worth. Furthermore, while small differences in GPA calculations can amplify differences in rank ordering, these crude snapshots of relative performance provide easy and popular comparability for nonspecialist publics. However, there is something particularly distinctive about the ranking of UK philosophy departments described above: it is based on an evaluation of the impact of their research. By impact in this context, one would ordinarily imagine journal citations and other demonstrable measures of quality within the field of academic philosophy. Such bibliometrics have attracted considerable attention from analysts (e.g., Gingras, 2016). Yet this would be wrong. Impact in this UK setting means the social and economic beneficial impact outside academia. In other words, the departments of philosophy at Birmingham, Keele, and elsewhere in the United Kingdom were graded and ranked in terms of the social and beneficial impact of their research. In fact, all subject areas in UK universities were evaluated for this kind of impact as part of a major evaluation of research quality, the Research Excellence Framework (REF2014 hereafter, which is the successor to the Research Assessment Exercises of previous decades). UK universities made 1,911 submissions across all subject areas from 52,061 staff who produced 191,150 research “outputs” 3 Playing and Being Played by the Research Impact Game","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121164130","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Index","authors":"","doi":"10.7551/mitpress/11087.003.0029","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0029","url":null,"abstract":"","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127776303","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PubPeer: Scientific Assessment Without Metrics","authors":"","doi":"10.7551/mitpress/11087.003.0015","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0015","url":null,"abstract":"","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"41 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133227518","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Global University Rankings: Impacts and Applications","authors":"","doi":"10.7551/mitpress/11087.003.0009","DOIUrl":"https://doi.org/10.7551/mitpress/11087.003.0009","url":null,"abstract":"","PeriodicalId":186262,"journal":{"name":"Gaming the Metrics","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132561153","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}