{"title":"How to rate programming skills in programming experiments?: a preliminary, exploratory, study based on university marks, pretests, and self-estimation","authors":"Sebastian Kleinschmager, Stefan Hanenberg","doi":"10.1145/2089155.2089161","DOIUrl":"https://doi.org/10.1145/2089155.2089161","url":null,"abstract":"Rating of subjects is an important issue for empirical studies. First, it is desirable for studies that rely on comparisons between different groups to make sure that those groups are balanced, i.e. that subjects in different groups are comparable. Second, in order to understand to what extent the results of a study are generalizable it is necessary to understand whether the used subjects can be considered as representative. Third, for a deeper understanding of an experiment's results it is desirable to understand what different kinds of subjects achieved what results. This paper addresses this topic by a preliminary, exploratory study that analyzes three different possible criteria: university marks, self-estimation, and pretests. It turns out that neither university marks nor pretests yielded better results than self-estimation.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130236257","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Using metaphors from natural discussion to improve the design of arcum","authors":"Macneil Shonle, W. Griswold, Sorin Lerner","doi":"10.1145/2089155.2089165","DOIUrl":"https://doi.org/10.1145/2089155.2089165","url":null,"abstract":"In this paper we present an exploratory pair-programming study aimed at investigating how programmers use a tool and language designed for performing crosscutting change tasks. Through a qualitative analysis of the pairs' discussions, we identify the metaphors that the participants used to think about crosscutting change tasks, which allowed us to infer their expectations. The metaphors of particular interest were the comparisons participants used to describe their approach in terms of other meta and non-meta programming tasks. From this analysis, we identified challenges the participants encountered in writing custom checks and refactorings.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124720384","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Categorization of concerns: a categorical program comprehension model","authors":"Tim Frey, Marius Gelhausen, G. Saake","doi":"10.1145/2089155.2089171","DOIUrl":"https://doi.org/10.1145/2089155.2089171","url":null,"abstract":"Program comprehension models lack associations with the paradigm of separation of concerns. We present a holistic program comprehension model based on categorization studies of psychology. A comparison of research about categorization and separation of concerns is used to develop the model. The cognition in this model is influenced by the context wherein a programmer investigates the code. The comprehension process starts with some ad-hoc concerns that are about to be refined by following an investigation strategy and a vertical process study. Through this study, the concerns refinement may imply an update on the knowledge and the adoption of a new behavior for the investigation strategy. Our model can serve as a starting point for further investigations.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125251013","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ankica Barisic, Vasco Amaral, M. Goulão, B. Barroca
{"title":"Quality in use of domain-specific languages: a case study","authors":"Ankica Barisic, Vasco Amaral, M. Goulão, B. Barroca","doi":"10.1145/2089155.2089170","DOIUrl":"https://doi.org/10.1145/2089155.2089170","url":null,"abstract":"Domain-Specific Languages (DSLs) are claimed to increment productivity, while reducing the required maintenance and programming expertise. In this context, DSLs usability is a key factor for its successful adoption.\u0000 In this paper, we propose a systematic approach based on User Interfaces Experimental validation techniques to assess the impact of the introduction of DSLs on the productivity of domain experts. To illustrate this evaluation approach we present a case study of a DSL for High Energy Physics (HEP).\u0000 The DSL on this case study, called Pheasant (PHysicist's EAsy Analysis Tool), is assessed in contrast with a pre-existing baseline, using General Purpose Languages (GPLs) such as C++. The comparison combines quantitative and qualitative data, collected with users from a real-world setting. Our assessment includes Physicists with programming experience with two profiles; ones with no experience with the previous framework used in the project and other experienced.\u0000 This work's contribution highlights the problem of the absence of systematic approaches for experimental validation of DSLs. It also illustrates how an experimental approach can be used in the context of a DSL evaluation during the Software Languages Engineering activity, with respect to its impact on effectiveness and efficiency.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"102 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115748173","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"How much information do software metrics contain?","authors":"J. Gil, M. Goldstein, Dany Moshkovich","doi":"10.1145/2089155.2089169","DOIUrl":"https://doi.org/10.1145/2089155.2089169","url":null,"abstract":"Software metrics computation and presentation are considered an important feature of many software design and development tools. The System Grokking Technology developed by IBM research enables investigation, validation and evolution of complex software systems at the level of abstraction suitable for human comprehension. As part of our ongoing effort to improve the tool and offer more useful abstractions we considered adorning the presented information with software metrics. The difficulty in doing that is in selecting among the legions of metrics competing for both scarce screen space and for the architect's attention. In this paper, we describe a new criterion for evaluating the competing metrics based on a normalized version of Shannon's information theoretical content. We also give values of these in a large software corpus and for a large set of metrics.\u0000 Based on our measurements and this criterion, we can recommend the presentation of two metrics: module centrality, as measured by a variant of Google's classical page ranking algorithm, and module size, as measured by Chidamber and Kemerer's WMC metric.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133519263","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Are object protocols burdensome?: an empirical study of developer forums","authors":"Ciera Jaspan, Jonathan Aldrich","doi":"10.1145/2089155.2089168","DOIUrl":"https://doi.org/10.1145/2089155.2089168","url":null,"abstract":"Object protocols are a commonly studied research problem, but there is little known about their usability in practice. In particular, there is little research to show that object protocols cause difficulty for developers. In this work, we use community forums to find empirical evidence that object protocols are burdensome for developers. We analyzed 427 threads from the Spring and ASP.NET forums and discovered that 69 were on a protocol violation. We found that violations of protocols result in unusual runtime behavior rather than exceptions in 45% of our threads, that questions took an average of 62 hours to resolve, and that even though 54% of questions were repeated violations of similar protocols, the manifestation of the violation at runtime was different enough that developers could not search for similar questions.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127528753","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Capturing and analyzing low-level events from the code editor","authors":"Youngseok Yoon, B. Myers","doi":"10.1145/2089155.2089163","DOIUrl":"https://doi.org/10.1145/2089155.2089163","url":null,"abstract":"In this paper, we present FLUORITE, a publicly available event logging plug-in for Eclipse which captures all of the low-level events when using the Eclipse code editor. FLUORITE captures not only what types of events occurred in the code editor, but also more detailed information such as the inserted and deleted text and the specific parameters for each command. This enables the detection of many usage patterns that could otherwise not be recognized, such as \"typo correction\" that requires knowing that the entered text is immediately deleted and replaced. Moreover, the snapshots of each source code file that has been opened during the session can be completely reproduced using the collected information. We also provide analysis and visualization tools which report various statistics about usage patterns, and we provide the logs in an XML format so others can write their own analyzers. FLUORITE can be used for not only evaluating existing tools, but also for discovering issues that motivate new tools.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130031839","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Staking claims: a history of programming language design claims and evidence: a positional work in progress","authors":"Shane Markstrum","doi":"10.1145/1937117.1937124","DOIUrl":"https://doi.org/10.1145/1937117.1937124","url":null,"abstract":"While still a relatively young field, computer science has a vast body of knowledge in the domain of programming languages. When a new language is introduced, its designers make claims which distinguish their language from previous languages. However, it often feels like language designers do not feel a pressing need to back these claims with evidence beyond personal anecdotes. Peer reviewers are likely to agree.\u0000 In this paper, we present preliminary work which revisits the history of such claims by examining a number of language design papers which span the history of programming language development. We focus on the issue of claim-evidence correspondence, or determining how often claims are or are not backed by evidence. These preliminary results confirm that unsupported claims have been around since the inception of higher level programming in the 1950s. We stake a position that this behavior is unacceptable for the health of the research community. We should be more aware of valiant and effective efforts for supplying evidence to support language design claims.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126602725","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"User evaluation of correctness conditions: a case study of cooperability","authors":"Caitlin Sadowski, Jaeheon Yi","doi":"10.1145/1937117.1937119","DOIUrl":"https://doi.org/10.1145/1937117.1937119","url":null,"abstract":"In order to find and fix concurrency bugs, programmers must reason about different possible thread interleavings -- context switches may occur at any program point, all with the potential for thread interference. To reduce the number of thread interference points to consider, the correctness criterion of cooperability ensures that code executes as if context switches may happen only at specific yield annotations. This paper provides empirical evidence that co-operability makes it easier to find concurrency bugs.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"87 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121719392","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Comparing the usability of library vs. language approaches to task parallelism","authors":"Vincent Cavé, Zoran Budimlic, Vivek Sarkar","doi":"10.1145/1937117.1937126","DOIUrl":"https://doi.org/10.1145/1937117.1937126","url":null,"abstract":"In this paper, we compare the usability of a library approach with a language approach to task parallelism. There are many practical advantages and disadvantages to both approaches. A key advantage of a library-based approach is that it can be deployed without requiring any change in the tool chain, including compilers and IDEs. However, the use of library APIs to express all aspects of task parallelism can lead to code that is hard to understand and modify. A key advantage of a language-based approach is that the intent of the programmer is easier to express and understand, both by other programmers and by program analysis tools. However, a language-based approach usually requires the standardization of new constructs and (possibly) of new keywords. In this paper, we compare the java.util.concurrent (j.u.c) library [14] from Java 7 and the Habanero-Java (HJ) [16] language, supported by our experiences in teaching both models at Rice University.","PeriodicalId":217446,"journal":{"name":"Workshop on Evaluation and Usability of Programming Languages and Tools","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2010-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122267910","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}