Open source code matches proprietary for quality

The quality of open source code is on a par with proprietary code, particularly in cases where codebases are of similar size.

This was one of the key findings of development testing company Coverity's annual Scan report. The company measured the quality of code by comparing defect density – the number of defects per 1,000 lines of code. The average defect density for the software industry is 1.0.

The report found that open source projects that had completed development testing using Coverity Scan had an average defect density of 0.45. This figure is based on analysis of 37 million lines of code from 45 of the most active open source projects in Scan.

In comparison, the average defect density for proprietary codebases was 0.64, based on analysis of 300 million lines of code from 41 proprietary codebases. The average proprietary codebase has 7.5 million lines of code, compared to 832,000 lines for open source projects.

“Really the results are pretty amazing,” said Zack Samocha, director of the Coverity Scan Project. “Open source projects are acting like commercial products. If I am a commercial company and I want to adopt open source, there are definitely projects out there that are trustworthy.”

Coverity's Scan report also looked in greater detail at Linux 2.6, PHP 5.3, and PostgreSQL 9.1 – three open source projects that are considered to have superior code quality and can be used as industry benchmarks. These projects achieved defect densities of .62, .20, and .21 respectively.

Samocha explained that these three projects had been chosen to give a representative cross-section of the open source market: Linux is an operating system, PHP is a language and PostgreSQL is a database. He said that all these projects have a very low defect density, because they all use tools like Coverity Scan to achieve quality.

“If you take PHP as an example, it's literally part of the process that they use Coverity on a daily basis every time they need to get the code to the right level,” said Samocha.

He added that the more mature projects now understand that code quality can give them a competitive advantage when customers are choosing which open source project to adopt.

“The quality of our code is critical to the ongoing success and adoption of PHP, which includes some of the world's most popular web sites,” said Rasmus Lerdorf, creator of PHP. “As our code grows and becomes more complex, Scan will become even more important for us as a way to help improve our code quality.”

Coverity Scan is a cloud-based offering for the open source community, which automatically tests source code for software defects that could lead to product crashes, unexpected behaviour, security breaches, or catastrophic failure. The company has more than 1,100 customers, including Adobe, Alcatel Lucent, Samsung and Vodefone.