Software assurance with samate reference dataset, tool standards, and studies

Abstract

Today's avionics systems depend more and more on software from many sources: vendors, subcontractors, in-house, and open source. System interactions are exposed to external agents in contexts from air-to-ground links to OS patches downloaded via the Internet. This is a huge amount of software with the risk of attack from distant global sites. Yet users need assurance that the software will work and not create security problems. We focus on NIST's Software Assurance Metrics And Tool Evaluation (SAMATE) project and its contribution. SAMATE is developing specifications, metrics, and automated test suites for software assurance tools. For instance, source code security analyzers can help developers produce software with fewer security flaws. They can also help identify malicious code and poor coding practices that lead to vulnerabilities. The project's publicly available reference dataset, the SRD, contains more than 1800 flawed (and fixed!) program examples to help evaluate software assurance tools and algorithms. These metrics and reference datasets help purchasers confirm tool vendors' claims. We also study the assurance impact of tool use, methods, and techniques.