Last week, BBC News reported that security firm Prevx will release a "small program" that will gather data about the effectiveness of anti-virus software:
In the face of the tidal wave of malware, said Mr Bolin, PC users need a better way to find out how well they are being protected and how long they have been at risk.
Mr Bolin believes the way to get a better sense of the performance of security companies is via a small program that sits on a PC and logs when files are installed.
The program would lie dormant most of the time but would alert a user if it noticed that a fix had been created for a particular virus or trojan it had spotted on a PC.
The idea seems to be that measuring the lag time between when a file is installed until a definitions file addresses it will "bring about change in an industry that is not changing."
My comments here are based on the BBC story, as I couldn't find any additional information about the plan at Prevx's website or blog.
The first problem with the plan is that Prevx is itself an anti-virus company. Therefore, it's hard to believe the real goal isn't to make its competitors look bad. Even if this isn't the case, the company's decision to release this software on its own, rather than working through a trade association or an independent third party, makes it appear to be the case. Only with independent scrutiny and a design that is intended to objectively measure effectiveness should a tool like this be considered reputable.
Another problem is that security software is changing. Most major AV products now use a combination of techniques to prevent and detect malware. They don't rely only on file signatures, but instead use various techniques for identifying suspicious behaviors, blocking the exploits that deliver the malware, and leveraging cloud-based data. Based on the BBC article's description of the Prevx software, it seems that the software would measure only those files that got past all of these defenses. This would measure not the products' effectiveness at protecting users, but only their response time to files that slipped through an extensive set of defenses.
My hope is that Prevx will reconsider this ill-conceived idea. Graham Cluley, from rival AV vendor Sophos, has his own critique of the Prevx plan here.
