{"title":"Taking the mystery out of experimental design-and a proposal","authors":"L. Votta","doi":"10.1109/METRIC.1998.731243","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731243","url":null,"abstract":"Summary form only given. As the computer science and software engineering community recognizes the importance of empirical studies, we look to many other disciplines for models of empirical work. At the core of many of these models are methods and techniques refined over the years that are known to be credible to the discipline's typical environments. One way of capturing this knowledge is to abstract the method into an experimental design, but what is an experimental design? I try to take some mystery out of experimental design. The key ideas are to recognize that science and engineering are founded upon credibility, which can range in empirical studies from the weak credibility of specific, non-verifiable, anecdotal reports to the strong credibility of experiments to show a casual relationship with that of the case study somewhere in the middle. The experimental design is the logic and interpretation infrastructure that attempts to show a causal relationship. I describe several experimental designs from different disciplines and abstract the key elements to form an operational definition. Finally, I observe that computer science and software engineering environments are similar to the biological sciences and conclude that an excellent match for our environments is the in vivo and in vitro empirical model and their experimental designs.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"157 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-11-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132028071","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Getting a handle on the fault injection process: validation of measurement tools","authors":"Sebastian G. Elbaum, J. Munson","doi":"10.1109/METRIC.1998.731238","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731238","url":null,"abstract":"In any manufacturing environment, the fault injection rate might be considered one of the most meaningful criterion to evaluate the goodness of the development process. In our field, the estimates of such a rate are often oversimplified or misunderstood generating unrealistic expectations on their prediction power. The computation of fault injection rates in software development requires accurate and consistent measurement, which translates into demanding parallel efforts for the development organization. This paper presents the techniques and mechanisms that can be implemented in a software development organization to provide a consistent method of anticipating fault content and structural evolution across multiple projects over time. The initial estimates of fault insertion rates can serve as a baseline against which future projects can be compared to determine whether progress is being made in reducing the fault insertion rate, and to identify those development techniques that seem to provide the greatest reduction.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"905 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116402382","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Using a Personal Software Process/sup SM/ to improve performance","authors":"W. Hayes","doi":"10.1109/METRIC.1998.731227","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731227","url":null,"abstract":"The use of software measurement and process definition by individual engineers is embodied in the Personal Software Process (PSP)/sup SM/, a collection of techniques and guidelines for individual software engineers to use in building software. This paper presents a brief overview of the PSP and summarizes data collected by engineers in order to illustrate the efficacy of the PSP. Implications for the use of methods associated with statistical process control are discussed, and the power of rigorous data collection by individual software engineers is highlighted through discussion of empirical data.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"105 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132113116","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Experiences with program static analysis","authors":"Hideto Ogasawara, M. Aizawa, Atsushi Yamada","doi":"10.1109/METRIC.1998.731235","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731235","url":null,"abstract":"Conventionally, software quality has been measured mainly by the number of test items, the test coverage, and the number of faults in the test phase. This approach of relying heavily on testing is not satisfactory from a quality assurance viewpoint. Since software is becoming larger and more complex, quality must be assured from the early phases, such as requirements analysis, design and coding. Code reviews are effective to build in software quality from the coding phase. However, for a large-scale software development, there are limitations in covering all the programs. The advantage of using static analysis tools is the capability to detect fault-prone programs easily and automatically. We describe the effective use of a static analysis tool, and show the effectiveness of the static analysis technique.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132501171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Software metrics decision support system","authors":"J. Ulvila, J. O. Chinnis, J. Gaffney","doi":"10.1109/METRIC.1998.731245","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731245","url":null,"abstract":"The software metrics decision support system (SMDSS) methodology was developed to address the problem of selecting the \"best\" set of values for quantifiable software project goals commensurate with various constraints and management's desires. The methodology was prototyped in a proof-of-concept tool, called MAUS, which uses multi-attribute utility theory (MAU). The MAU-based decision-making methodology is designed to aid software and systems managers in the use of a systematic, objective, and documentable quantification method. MAUS implements some novel aspects and extensions of the MAU methodology. MAUS uses MAU to explore a \"space\" of possible choices, and it utilizes a constraint equation to generate alternatives. The SMDSS methodology and the supportive MAUS tool promote implementation of the guidance provided in Practical Software Measurement: A Guide to Objective Program Insight.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"58 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116499611","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Coupling metrics for object-oriented design","authors":"R. Harrison, S. Counsell, R. Nithi","doi":"10.1109/METRIC.1998.731240","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731240","url":null,"abstract":"We describe and evaluate some recently innovated coupling metrics for object-oriented (OO) design. The Coupling Between Objects (CBO) metric of Chidamber and Kemerer (1991) is evaluated empirically using five OO systems, and compared with an alternative OO design metric called SAS, which measures the number of associations between a class and its peers. The NAS metric is directly collectible from design documents such as the Object Model of OMT. Results from all systems studied indicate a strong relationship between CBO and NAS, suggesting that they are not orthogonal. We hypothesised that coupling would be related to understandability, the number of errors and error density. So relationships were found for any of the systems between class understandability and coupling. Only limited evidence was found to support our hypothesis linking increased coupling to increased error density. The work described in this paper is part of the 'Metrics for OO Programming Systems' (MOOPS) project, which aims to evaluate existing OO metrics, and to innovate and evaluate new OO analysis and design metrics, aimed specifically at the early stages of development.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132289880","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A methodology for evaluating predictive metrics","authors":"J. Rosenberg","doi":"10.1109/METRIC.1998.731244","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731244","url":null,"abstract":"For over thirty years, software engineers have been interested in the ability to accurately measure characteristics of software and its production which could lead to improvements in both. In that time, a large number of metrics have been proposed, some with attempts at empirical validation of their effectiveness. Unfortunately, many if not most of these laudable efforts at empirical validation have foundered on a lack of knowledge about the appropriate methods to use. For example, a central goal in software metrics is the prediction of software characteristics based on other metrics of the software or its production process. This prediction problem is a quintessentially statistical one, but the lack of statistical training in the typical crowded engineering curriculum leaves most engineers uncertain about how to proceed. The result has been many well-intentioned but poorly executed empirical studies. This paper addresses this problem by providing a simple methodology for the predictive evaluation of metrics.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129502242","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
K. Emam, Jean-Martin Simon, Sonia Rousseau, Eric Jacquet
{"title":"Cost implications of interrater agreement for software process assessments","authors":"K. Emam, Jean-Martin Simon, Sonia Rousseau, Eric Jacquet","doi":"10.1109/METRIC.1998.731225","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731225","url":null,"abstract":"Much empirical research has been done on evaluating and modeling interrater agreement in software process assessments. Interrater agreement is the extent to which assessors agree in their ratings of software process capabilities when presented with the same evidence and performing their ratings independently. This line of research was based on the premise that lack of interrater agreement can lead to erroneous decisions from process assessment scores. However, thus far we do not know the impact of interrater agreement on the cost of assessments. We report on a study that evaluates the relationship between interrater agreement and the cost of the consolidation activity in assessments. The study was conducted in the context of two assessments using the emerging international standard ISO/IEC 15504. Our results indicate that for organizational processes, the relationship is strong and in the expected direction. For project level processes no relationship was found. These results indicate that for assessments that include organizational processes in their scope, ensuring high interrater agreement could lead to a reduction in their costs.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"50 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124051358","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Applying software metrics to formal specifications: a cognitive approach","authors":"R. Vinter, M. Loomes, D. Kornbrot","doi":"10.1109/METRIC.1998.731248","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731248","url":null,"abstract":"It is generally accepted that failure to reason correctly during the early stages of software development causes developers to make incorrect decisions which can lead to the introduction of faults or anomalies in systems. Most key development decisions are usually made at the early system specification stage of a software project and developers do not receive feedback on their accuracy until near its completion. Software metrics are generally aimed at the coding or testing stages of development, however, when the repercussions of erroneous work have already been incurred. This paper presents a tentative model for predicting those parts of formal specifications which are most likely to admit erroneous inferences, in order that potential sources of human error may be reduced. The empirical data populating the model was generated during a series of cognitive experiments aimed at identifying linguistic properties of the Z notation which are prone to admit non-logical reasoning errors and biases in trained users.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128705398","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Birk, Pieter Derks, D. Hamann, J. Hirvensalo, M. Oivo, Erik Rodenbach, R. Solingen, J. Taramaa
{"title":"Applications of measurement in product-focused process improvement: a comparative industrial case study","authors":"A. Birk, Pieter Derks, D. Hamann, J. Hirvensalo, M. Oivo, Erik Rodenbach, R. Solingen, J. Taramaa","doi":"10.1109/METRIC.1998.731234","DOIUrl":"https://doi.org/10.1109/METRIC.1998.731234","url":null,"abstract":"In ESPRIT project PROFES, measurement according to the Goal/Question/Metric (GQM) approach is conducted in industrial software projects at Drager Medical Technology, Ericsson Finland, and Schlumberger Retail Petroleum Systems. A comparative case study investigates three different ways of applying GQM in product-focused process improvement: long-term GQM measurement programmes at the application sites to better understand and improve software products and processes; GQM-based construction and validation of product/process dependency models, which describe the process impact on software quality; and cost/benefit investigation of the PROFES improvement methodology using GQM for (meta-) analysis of improvement programmes. This paper outlines how GQM is applied for these three purposes.","PeriodicalId":444081,"journal":{"name":"Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262)","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1998-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125007843","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}