“Enterprise users experienced an average of 339 Web malware encounters per month in 4Q11.” (up 205% year over year)

Avg. 20,141 unique Web malware hosts per month in 2011 (vs. 14,217 in 2010)

Source: Cisco 4Q11 Global Threat Report (Jan. 2012)
*****
Approx. 30,000 new malicious URLs each day in 2H11; 80% of those are legitimate

85% of malware comes from the web

Source: Sophos Security Threat Report 2012 (Jan. 2012)
*****
Malicious sites up 240 percent in 2011

40% of malnet entry points are via search engines/portals

Source: Blue Coat Systems 2012 Web Security Report (Feb. 2012)
*****
23% of malicious domain registrations could be blocked with basic validation of contact info

Source: Abused Internet Domain Registration Analysis for Calculating Risk and Mitigating Malicious Activity by KnujOn.com (Feb. 2012)
*****
Rogue AV campaign infected 200,000 Web pages, 30,000 unique hosts; more than 85% of sites in US, but more geographically dispersed visitors.

Source: Websense via Dark Reading (Mar. 2012)
*****
On average, two popular websites (among the Alexa top 25,000) serve drive-by downloads each day.

An estimated 1.6 million vulnerable users were exposed to drive-by downloads in one month across 58 popular (Alexa top 25,000) sites.

Source: Barracuda Labs (Mar. 2012)

We’re in the early stages of what we expect to be a three month pilot. So far, there are a lot of unanswered questions. Here are a few of the big ones:

• What will the inaugural set of tools/services look like? So far, we’re thinking of a data exchange API and a basic Web interface for searching the data.
• Who will have access to the data? Those with the best data often have valid (and occasionally not-so-valid) reasons for not wanting to share their data openly. We want to offer flexibility that encourages broad sharing but allows more limited sharing where appropriate. So, we’re imagining some sort of tiered permissions model.
• What incentives will there be to contribute data? Two models I’ve seen used before are quid pro quo—you earn access equivalent to what you contribute—and ”minimum threshold,” in which you must contribute a certain amount, after which you get full access. Both of these could have value, but it would be nice to provide access to a broader audience than just those who have substantial data to contribute.
• Which database platform should we use? Right now, our developer, Matthew, is experimenting with MongoDB (using Java for the middleware layer that will manage the data).

We’ll do our best to blog periodically throughout the pilot as we refine our answers to these and many other questions. Meanwhile, we’d love to hear your suggestions and other feedback in the comments or via email (contact <at> ourdomain).

LeaseWeb has a press release available here that outlines some of the details of our partnership. Don’t forget to Like our Facebook page for info on new partnerships, security news, and tips for webmasters and Internet users!

During our most recent Partners Forum call, we had an animated discussion related to these two questions. Our conversation covered a lot of ground, but here are a few key points that came up:

• There is substantial variation in how registries and registrars see their own roles. Some disavow any responsibility for addressing malicious name registrations. Others are much more hands-on.
• Registries and registrars come in all shapes and sizes. Smaller ones may need tools or support to manage abuse effectively.
• Often, for those reporting malicious URLs/sites, it’s the hosting providers rather than the registrars/registries that are the best first point of contact. (Though in some cases, the hosting providers are the registrars.)
• Registrars/registries have understandable concerns about being overzealous in shutting down domains. It’s easier to justify takedowns of harmful code than undesirable/illegal content, and of purely malicious domains than compromised domains. Registrars and registries need tools and data sources that help increase their confidence in differentiating between these cases.
• Takedowns are not the only remedy. Education of customers (in cases of compromise) can be a valuable role for registrars/registries (possibly in collaboration with StopBadware or other parties).

It’s clear that we have not definitively answered our two questions, but we’ve come up with great fodder for further discussion and action. If you want to be part of the conversation, we’re always looking to add new Partners.

I served on the group that developed the Code, the elegantly named CSRIC III Working Group 7. (Last week, I explained why fighting botnets is critical to StopBadware’s mission to make the Web safer.) The finished product is a testament to the collaborative spirit of the group’s members and the fearless leadership of the group’s chairman, Mike O’Reirdan.

There is, of course, room for criticism of the Code. I was, for example, disappointed that telling customers “go to this website to check if we’ve found bot traffic from your IP address” is considered a valid form of customer notification. The lack of any formal system to track which ISPs have agreed to adopt the Code (let alone verify that they’re actually following it) is also frustrating. If it had been up to me, I also would have more closely mimicked Australia’s model, which supplements the code of conduct with a national data clearinghouse of bot detection data.

Still, with all these complaints, we should consider the Code a step forward. With broad support (and substantial early adoption) from the ISP industry, it’s clear that millions of U.S. consumers will soon have more information to help them prevent badware, to learn if their devices are infected, and to assist them in cleaning their devices up. And the Code’s requirement that ISPs share information should help drive improved measurement and better anti-bot strategies.

Several groups, including MAAWG, the Industry Botnet Group, OTA, and even CSRIC III Working Group 7 continue to build upon the work done to date. There’s still plenty to be done, but it’s great to see so much movement in the right direction.

At StopBadware, we have a list of URLs that community members have reported to us as containing badware. We manually test all URLs from this feed to see if they contain badware, and when badware is present, we report the URLs to appropriate parties. In July, I started reporting URLs from the community feed in accordance with StopBadware’s Best Practices for Reporting Badware URLs; I tracked responses and regularly checked back to see if the sites had been cleaned up or taken down.

In October 2011 I began an academic study based on StopBadware’s pilot reporting project. My methodology was as follows: On day 0, I manually tested a URL taken from StopBadware’s community feed. If it was actively delivering badware, I randomly assigned the URL to one of three groups: control, minimal, and full. For the control group of URLs, no reports were sent out. For the URLs assigned to the minimal group, I sent out badware reports to the appropriate parties, but the reports contained only a minimal amount of information*. For the URLs assigned to the full group, I sent out minimal reports with additional detailed information* at the end. After the reports were sent out, I followed up on each of the URLs 1, 2, 4, 8, and 16 days after the day that I first found badware (day 0) to see if that badware had been removed.

The table below shows the probability that a URL will be “permanently” cleaned up after so many days. For the purposes of this study, I considered a URL “permanently” cleaned up on a day if on this day and every future follow-up day the URL was clean.

*percentages represent the probability that a URL is “permanently” clean after x days with the specified level of reporting.

As you can see, sending a full report substantially improved the likelihood that an infected URL would be cleaned up. Full reports were also observed to be significantly more effective than minimal and no reports on every single day that I followed up on a URL.

But what does this all mean? It means that sending a detailed badware report appears to be an effective measure for getting a badware URL cleaned up. Furthermore, providing more details seemed to be helpful to the site owners and abuse teams who had the ability to clean up the badware.

We’re currently working on ascertaining whether other forms of notification sent in the same time frame (e.g., malware notifications from Google Webmaster Tools) could have prompted some of the badware URL clean-up we observed. Tyler Moore and I are in the process of writing an academic paper with the complete methodology and full results of this study; the paper will be published later this year.

*For examples of minimal reports and additional information, please see pages B-2 to B-4 of StopBadware’s reporting best practices.

Much of the work from these groups so far has been focused on end user desktops, which comprise the vast majority of bot devices. (There have been examples of mobile phone and Web-based botnets, as well, but they’re not commonplace.) It is hoped that, by leveraging relationships with their customers, ISPs and other companies can help users prevent badware infection and more quickly identify and remove infection. This would decrease the availability of bots available to malicious actors.

So, how does cleaning up the desktops relate to cleaning up the Web? Well, botnets are used to find and compromise vulnerable websites. They are also used to send or post spam that directs users to click links to badware sites. And the money earned through the use of botnets helps support the same underground economy that drives malicious activity on the Web. Reducing the strength of botnets, then, can help reduce the problem of badware websites, too.

Conversely, badware websites help drive the problem of botnets. In particular, malicious sites are one of the main vectors by which end users encounter the malware that turns their PCs into bots. Some botnets also receive their instructions from Web-based command and control servers. Therefore, cleaning up the Web is all about reducing the threat of botnets.

StopBadware focuses, first and foremost, on the prevention, mitigation, and remediation of badware websites. We must, however, avoid tunnel vision and continue to support complementary efforts that are integral to making the Web safer for all of us.

Our conversation with Commtouch resulted in our designing a joint survey aimed at site owners; from November 2011 to the end of January 2012, we offered the survey, through a variety of outlets, to webmasters whose sites had been compromised. At the end of the survey period, we had collected responses from more than 600 webmasters who provided us with stories about their experiences. Today, StopBadware and Commtouch published a new report based on this survey data: Compromised Websites: An Owner’s Perspective highlights webmasters’ struggles with hacked sites and presents statistics and opinions from site owners.

A few highlights from the survey:

• About half of the site owners surveyed discovered the hack when they attempted to visit their own site(s) and saw a browser or search engine warning.
• 26% of the respondents had not figured out how to resolve the problem at the time they completed the survey.
• 40% of site owners changed their opinion of their web hosting provider following a compromise.