Microsoft-sponsored report slams Linux security
Are independent reports meaningful any more?
By John Dunn | Techworld | Published: 14:10, 23 March 2005
An “independent” report that claims Linux security vulnerabilities are more numerous and severe than in Windows has been confirmed as having been funded by Microsoft.
The Role Comparison Report report by Richard Ford of the Florida Institute of Technology's College of Engineering, and Herbert Thompson of security company Security Innovation, was originally previewed in draft form at the RSA conference in February, where it attracted inevitable criticism for its methodology and claimed bias.
The study set out to compare Windows Server 2003 and Red Hat Enterprise Linux ES3, running a range of applications atop the operating systems to check their ability to secure a web server setup. The team then compared the number of known vulnerabilities for the two, finding 52 for Windows, 174 for a default Linux server install, and 132 for a bare-bones Linux setup.
The team found that Windows also beat Linux using the “days of risk” measurement – how long it took a vendor to issue a fix for a vulnerability after it had become publicly disclosed – with an average of 31.3 days against Linux’s 71.4, or 69.6 for the minimal install.
After each of these vulnerabilities had been accorded a severity rating, Linux again scored poorly. During 2004, Windows Server 2003 had 1,145 of these rated as “high severity”, while even the minimal version of Red Hat Linux had almost double this number, at 2,124.
The published report (pdf) now confirms that its funding did indeed come from Microsoft, which is bound to undermine its credibility in the eyes of some. The authors counter this, noting, “We have full editorial control over all research and analysis presented in this report. We stand behind out methodology and execution of that methodology to determine objective results that will be useful to customers and security practitioners.”
The report has already been criticised by Mark J. Cox of Red Hat, who comments on it in his blog of this week, saying “Red Hat was not given an opportunity to examine the Role Comparison Report or its data in advance of publication and we believe there to be inaccuracies in the published "days of risk" metrics. These metrics are significantly different from our own findings based on data sets made publicly available by our Security Response Team.
Last year, a report from Forrester came up with similar conclusions to those of the Role Comparison Report, finding that between 1 June 2002 and 31 May 2003, Windows was vulnerable for fewer days than Red Hat, Debian, MandrakeSoft and SUSE Linux distributions.
What no report can do, however, is compare the risks faced by companies running the rival systems in real-world conditions. That would mean taking account not only of noted vulnerabilities and patching cycles but the likelihood of an attacker successfully targeting any one of them during the window of vulnerability. There is no evidence that one server operating system is more likely to be targeted than an other, so much of the “days of risk” hypothesis remains just that.
And with the industry and its appointees now turning out reports the independence of which is increasingly being questioned, even valuable information now risks getting lost amidst accusation and counter-accusation.