Every so often we get requests from readers asking us about comparisons between the different anti-virus products. These requests range from recommendations on how to compare oneself over to ready made comparison reports.

Virustotal

Typically we tend to use virustotal output in a lot of the diaries we write as it gives a good overview where a given malware is detected and how the different vendors named it. E.g:

Obviously some vendors are absent from these results.

Virustotal keeps some limited statistics online, but they're not useful in comparing products.

Build your own

Now if you collect enough of these you might build your own statistics on which product detects things you encounter better than the competition. It's not easy to collect enough of them to get a statistically significant sample, so running 2 or more of your favorite scanners in-house might be easier to get more significant results -but more limited in scope-.

Getting enough malware to scan could be done using proxy logs, stripped email attachments etc. Do take care with local privacy rules/laws before doing this!

3^rd Party Reports

There are some reports available about 3^rd party testing of anti-virus products.

• www.av-comparatives.org: updated every so often, includes a rating system.
• www.pcworld.com: article, more than a year old.
• www.av-test.org: runs out of a German University, not updated recently
  
• www.virus.gr: last updated in August 2006
• Consumer Reports has a comparison of 12 anti-virus products (subscribers only), it did get heavy negative feedback from the anti-virus vendors who seem not to like being put to the test.
• Virus bulletin has a report online for registered users and is referenced by many of our readers.
• Some more comparisons can be found at  antivirus-software.topchoicereviews.com and  www.consumersearch.com

Criteria

What's important to evaluate anti-virus products on? A test with a well known fake virus to see if it is detected (eicar), just will not expose the strengths and weaknesses of the different products and allow us to make a choice. Depending on the specific situation, we can be interested in:

• Few false positives: detecting known good software as malicious and crippling systems as a result has happened before, the impact of recovering from this should not be underestimated. While looking forward is hard, the hindsight view might tell it's own tale
  
• Few false negatives: not detecting malware is a bad thing, but it does happen by default in a technology that is basically reactive and where those creating the malware actually test their contraptions against the anti-virus products to make sure they are not detected at the time they release them.
• Timely signature updates: signature updates is the main vector anti-virus software uses to fix the above problems. The faster they are released the more protection you get as a customer.
• In corporate settings we want excellent centralized management. This should at least include a report that points us to individual machines not updating their definitions at all or in a timely manner. Ideally it does this without blocking signature updates when the roaming laptops are not connected to the corporate network or a VPN back to the office.
• Few vulnerabilities: Vulnerabilities in our security solutions are somewhat of a nightmare as they not only fail in their goal of making us more secure but also introduce more security problems.
• We do want variety if possible so that we use different engines in the different roles by e.g. using a different vendor on our email infrastructure and the desktops. The same goes for desktops and file servers.
• Ease of use.
• Price
• ...

With thanks to epablo, Vincent,  Bryan, William, and many others for contributing to this diary

--
Swa Frantzen -- NET2S