ConsumerReports Creates 5,500 Viruses

We hadn't seen any independent evaluation of antivirus software that measured how well products battle both known and new viruses, so we set out to fill that gap.

We tested a comprehensive selection of antivirus software for consumers in ways that accurately reflect real-world conditions, enlisting the help of consultants at Independent Security Evaluators (ISE), a computer-security consulting firm.

Together, we ran the 12 popular products in our Ratings through a grueling, custom-designed obstacle course.

There are several things here that do not seem right:

1. It is claimed that created viruses were "the kind you'd most likely encounter in real life" which is, of course, something the testers cannot know.

2. Creating new viruses for the purpose of testing and education is generally not considered a good idea - viruses can leak and cause real trouble (you can read an open letter on the AVIEN site about that).

3. There is a more scientific way of measuring real proactive detection of AV products on future malware - it is called "proactive testing" or "retrospective testing". The idea is to measure, say, 3-month old AV product against real field viruses that appeared within these last 3 months. The discussion of the methodology of such tests can be found here and some real test results with common AV products are on the AV-comparatives.org site.

There are some in the antivirus community that are appalled at what they believe to be shoddy work.

Publications need to use industry-standardized methods for testing. Organizations like Virus Bulletin have been doing this for years. Why can't publications follow their lead?

'This is likely to be a very unpopular move,' said John Hawes, Technical Consultant at Virus Bulletin. 'The AV community has always been very strongly opposed to the creation of new malware for any purpose. There's just no need for it - plenty of new viruses are being written all the time, why would anyone in a responsible position want to add to the glut?'

"When I read about what ConsumerReports has done I want to bash my head against a brick wall."