Spam filters block legitimate email, finds test
Trial Virus Bulletin tests point to problems...
By John Dunn | Techworld | Published: 15:33, 20 March 2009
Many anti-spam products still block an inconvenient amount of legitimate email, a new test of leading products has suggested.
The data emerged from preliminary tests carried out by the UK-based outfit Virus Bulletin (VB) using a methodology still being worked on by the company.
Although none of the eight products looked at are mentioned by name in the published results - reckoned to be unfair given the trial nature of the tests - all blocked between 0.4 percent (the worst performing) and 0.04 percent (the best performing) of legitimate ‘ham' emails.
The qualification is that the length of time the different products ran did vary from only three days to ten days, making comparisons impossible, but none appeared to be able to improve on this modest performance. For perspective, a false positive rate of even 0.04 percent over seven days of testing was on one product under test was equivalent to blocking 653 legitimate emails by VB's definition, or 93 per day.
"The emails concerned have been scrutinised and the majority proved to be emails from mailing lists and newsletters - email that is sent in bulk and notoriously difficult for filters to distinguish from spam," said the VB testers. "Of course, being difficult for the filters to distinguish is not an excuse for filters to block the mails."
The ability of the products to block genuine spam was only slightly more convincing, varying from 89.5 percent to 95.59 percent. A product achieving a basic ‘silver' stamp (above which lies Gold and Platinum) would need to achieve a spam-catch rate of 85 percent or above, with a false positive rate lower than 0.25 percent.
"We haven't mentioned the product names this time because this was a trial run - vendors submitted their products to us free of charge on the understanding (because the methodology was previously untested) that the published results would be anonymised (the vendors themselves have been supplied with their own test results)," said Virus Bulletin's Editor, Helen martin.
After announcing its intention to create an anti-spam test methodology in January, VB now says it plans to finalise a full run in April, with the results to be published in May. These will relate vendors to their full scores without the cloak of anonymity for protection.
