After months of looking into the infections of AS21844 (ThePlanet) we've decided to wrap up our investigations for now.  We have learned quite a bit from our communications with customers at ThePlanet.  While no one from ThePlanet has spoken with us officially we have learned that they possess a direct feed of infected URLs from Google.  This means that large customers of ThePlanet, such as HostGator, should have the ability to learn of infections directly from their provider.  Also partners such as Skenzo should be able to use the same list to purge previously infected, and now abandoned, domains from their monetization framework.  

 

WebsiteWelcome	7909
Skenzo FZE	2838
Unidentified	683
Site5 LLC	430
Bahram Boutorabi	192
SiteGround.com Inc.	171
webserver-a-rackshack.directi.com	166
Mochanin Corp	136
maktoob.com	119
server sea	115
Payam Torkian	115
Our Internet_ Inc	112

 

 

count	domain
10	vadakarapally.org
12	attorney2traffic.org
16	e-sense.tv
17	niftysensex.com

HostGator has roughly 7,563 unique infected domains according to our last count and ThePlanet has 20,298 unique infected domains with their true number likely around 17,000 (adjusting for Skenzo).  Where does that put ThePlanet in the context of our top 50 infected networks?  Exactly where they are now actually. The next closest network is GoDaddy's AS26496 with 11,576 infections.

A reader asked a question in the comments of the previous blog post about ThePlanet regarding the distribution of infections.  The reader wanted to know if the rest of the infections not attributed to Skenzo and HostGator were evenly distributed with less than 10% of the total infections.  The type of distribution the reader was describing is called a Power Law distribution.  A power law distributed population will look something like this: courtesy of Wikipedia.org

For this blog post I'm using data pulled from early May 2010 on AS21844 (ThePlanet) and find the infection counts are roughly power law distributed.  I've gone over the methodology to obtain this data in previous posts but it bears mentioning that I am using data distributed by RWhois organization names.  Later in this post I will look at the same data distributed by only IP address so that it can be compared with other AS blocks.  The raw infection counts look like this in graphical format:
The shape is precisely the same and it is obvious that there are a lot of organizations that have only single and double digit infections attributed to them.  The area between 500 and 2500 is entirely barren with only a single entry beyond 2500.  One of the issues when looking at data like this is the blur of data points below the 500 marker.  One could simply strip away the outliers (those data points above 500) but in this particular case I don't think that is an effective way to view the data.  In statistics people often "transform" the data to deal with this situation.  This generally means they divide all the numbers by some constant which allows the data to retain the same shape but become easier to read.  I generally favor the log/log method which means I take the log of each number and graph it that way.  Log (or logarithm) is a mathematical function best explained by Wikipedia but best thought of as a number "reducer" that can be applied uniformly across data.

To get a sense of the scale the log of 2500 is 7.8, the log of 500 is 6.2, the log of 100 is 4.6 and the log of 1 is 0.  Once the data is transformed we can see there is a little variance in the actual distribution but the fact that the line is sloping downward like that is another very good indicator of the power law distribution.

Last month we started an investigation into the massive numbers of infections we saw on ThePlanet’s AS21844 network. Last week we discovered the Rwhois server at ThePlanet and were able to get a more fine grained view of the infection distribution. 10% of the infections were attributed to Skenzo while 40% were attributed to HostGator resellers.
The infections we thought were attributed to Skenzo turned out to be abandoned badware domains. We think this problem will largely work itself out as Skenzo has no interest in monetizing from domains marked as badware.
The infections at HostGator were a bit more challenging. I communicated with several members of the HostGator team over the course of the last few weeks. They voiced some valid complaints that I will talk about in this post. The most important of which is the way infections are counted.
One domain, nyalines.com, had something like 1000 infections attributed to it. This is pretty unusual for our data partners to do. If there are more than a handful of infections at the same domain they will usually just list the entire domain. When we asked Google, the data partner responsible for that particular listing, they said the automated system they have in place thought it was better to list it that way.
Here is a sample of what they were talking about:

    10 vadakarapally . org
    10 websitecoders . org
    11 e-sense . tv
    11 malayalamwallpapers . net
    12 attorney2traffic . org
    15 kingvip . com
    17 niftysensex . com
    18 fountain.fountaintips . com
    19 findluxurywatch . com
    19 freewallpapershere . com
    20 quitsmokingtips4u . com
    21 shorthandlogic . com
    23 dir10 . net
    91 freenewdownload . com
   116 moviemark.com . br
   987 nyalines . com

We didn’t get any further explanation from Google so I am at a loss for why there was a need to mark the same domain 1000 times. The senior security tech at HostGator I spoke with felt that our report unfairly characterized HostGator and I would like to address that. We at StopBadware simply follow the data. We take what is in front of us and interpret as best we can for public consumption. When we are shown errors in our methodology we adapt it. Figuring out how to more accurately represent infections on the Internet is a giant part of what I do and over counting of a particular domain will be at the top of my list (along with Rwhois resolution). However ThePlanet is still at the top of the infection charts for US based web hosting providers. And even if we count each domain only once HostGator resellers accounted for 6655 of the infections within that network. I am very grateful for their team’s willingness to work with us to eradicate those infections.
It also bears mentioning that I don’t particularly think Google did anything wrong here either. They produce a list of URLs believed to contain badware on it and release it to their partners. We made the move to quantify this list so we could get some sense of whether things were getting better or worse. Both in terms of overall infections and infections within particular networks. Those metrics allow us to prioritize hubs of infection on the Internet and spend our scarce resources attacking where it counts.
We will begin the bulk appeal process to get the URLs HostGator has cleaned unmarked as badware. With some luck the high numbers of infections on AS21844 will start coming down.

I’ve been working on my investigation of ThePlanet and have some new and interesting results.

Skenzo has some valid concerns. They monetize abandoned domain names and apparently inherited a bunch of abandoned badware URLs. When Google rescans a site on its badware list and finds that the contents have disappeared or changed dramatically, Google does not necessarily assume that the site is clean. Which is to say that someone who simply deletes the page and doesn’t request a review might stay on the list for a prolonged time. The logic, I guess, is that they are preventing someone from simply deleting the page until they are cleared and then reinstate the previous content.
Skenzo did some investigating of their own with a list of URLs I provided them. They found the following:

• 635 URLs had not been visited by Google in the last 90 days
• 108 URLs Google had visited but did not find a suspicious page in the last 90 days
• 473 URLs marked as suspect in the last 90 days. This would be at the previous network and not on Skenzo’s infrastructure

There are obvious issues with Skenzo’s situation. Skenzo doesn’t want the badware URLs in their monetization network anyway so I introduced Skenzo to the Google team in the hopes that Google will just send them updated lists for removal. So that may have a happy ending.

WebsiteWelcome is a whole other headache. Earlier I only ran the top 50 IP addresses from the infections in AS21844. This means I excluded the “tail” of the distribution. Usually the tail is made up of small websites with 1-5 infections on their IP address. However what I didn’t realize at the time was that WebsiteWelcome is, quite literally, HostGator. I had assumed they were just a reseller but they seem to be the private label name used by all Host Gator resellers. So when I reran the entire list of infections in AS21844 through the RWhois server I got this result:

WebsiteWelcome 8317
Skenzo FZE 2592
No Orgname 474
Site5 LLC 389
SiteGround.com 205

This means that of ThePlanet’s 20,000 infections HostGator (under the WebsiteWelcome name alone) comprises ~40% of them. Those infections are spread out across 2,800 IP addresses. That is a really large percentage considering many of the top malware network lists have ThePlanet at the top. Worse I don’t have any way of making the list more granular. HostGator and I have been in touch via email but they refuse to go on record. I continue to send them URLs and they are working on cleaning up these hosts so far as I can tell.

[Update 4/27: Edited the part about Google’s policy for improved accuracy.]

Last week I wrote about infections which seem to plauge web host ThePlanet. A lot of information has come in since then that explains a bit of what is going on. First I have not received any official communications from ThePlanet regarding the infections. If someone at the this company would like to talk to us about how to incorporate this data into their abuse fighting efforts we’d love to help. I am sure there are others in this field who would offer the same.

Secondly we are not the only ones who found this problem percolating at ThePlanet. Intrepid reporter Brian Krebs pointed to research at FIRE that shows ThePlanet at the top of their most infected list. He goes on to say that a majority of the other badware trackers out there feel the same.

Lastly I was tipped off to the RWhois server at ThePlanet to allow a finer grain of resolution on the infections. I ran the top infections through and the results are eye opening. Two clients stand out as the majority source of issues.

IP Address	Infected	RWHOIS org name
174.120.120.151	2360	Skenzo FZE
74.54.82.209	704	Skenzo FZE
174.123.118.242	701	WebsiteWelcome
209.85.84.167	501	Skenzo FZE
74.54.82.151	294	Skenzo FZE
209.85.51.171	187	Skenzo FZE
70.85.203.98	154	Bahram Boutorabi
66.98.226.63	107	unknown
174.132.114.66	104	WebsiteWelcome
209.62.72.250	102	Skenzo FZE
74.54.82.223	90	Skenzo FZE
66.98.145.18	82	unknown
70.84.243.130	72	WebsiteWelcome
174.133.93.58	72	server sea
74.55.113.68	63	Payam Torkian
74.55.100.8	60	Skenzo FZE
75.125.230.50	59	007express.com
74.55.26.91	57	Fakhreddin
174.132.194.9	56	WebsiteWelcome
74.54.62.162	53	Xieno
74.53.162.242	52	xpower.net
74.52.114.250	52	brian bennett
74.52.105.66	48	WebsiteWelcome
67.19.92.170	48	PQC Service, LLC
67.19.140.10	44	NV Avid Corp.
209.85.51.176	44	Skenzo FZE
74.52.111.226	42	Websouls
70.86.72.202	40	hub4host
209.62.105.19	39	Skenzo FZE
75.125.148.76	36	4.CN
209.62.55.197	34	Payam Torkian
67.15.126.22	33	unknown
174.123.249.210	33	WebsiteWelcome
209.85.84.165	32	Skenzo FZE
75.125.198.122	31	i4serv
74.54.131.210	31	WebsiteWelcome
74.53.241.66	31	WebsiteWelcome
74.52.142.66	30	WebsiteWelcome

I’m told, but have not confirmed that Skenzo is a domain parking service and WebsiteWelcome is somehow associated to HostGator. If anyone from these two organizations would like to talk I’m here to help. I plan to use some of our historical data to chart how the infections grew in these two organizations and see if they correlate with any other security events. I’m hoping to see something like a spike around the time of a PHP bug or something similar.

I’ve also thought long and hard about how I would advise hosting firms like ThePlanet if I were in a position to do so. My current opinion, always subject to change, is that a graduated response should be used. Notify the client first in cases of extreme infections with a week of waiting. If there is no contact in a week then pull the sites offline (unplug the network connection only not the power) until the client makes contact. Then allow only IP addresses they designate through to allow them to clean up the server but not allow future infections to occur. If there is no contact at all within some maximum amount of time then keep the box offline until it occurs.

This is not something I’m ready to defend yet so if there are suggestions or comments I’m totally open to hear them. But it reflects a mixture of the best ideas I’ve heard from friends and colleagues about the situation.

EDIT: added line breaks to data section to make it readable

Earlier this month I started investigating the infections which continue to plague ThePlanet. As you can see in the chart I linked to they have sustained over 10,000 infections for several months now. In my research I found that many of these infections are not new and have been present on ThePlanet’s network going back as far as September 2009.

I wrote some code that performs a simple intersection test on two lists. That is to say, if you have list A and list B this program will tell you the amount of items that exist in both lists. I used a list of infected urls pulled from 3/1/2010 and compared it to lists of infected urls going back to September 2009. Here are the results I found:

[03/2010 and 02/2010] = 12,061
[03/2010 and 01/2010] = 10,417
[03/2010 and 12/2009] = 7,701
[03/2010 and 11/2009] = 6,129
[03/2010 and 10/2009] = 4,597
[03/2010 and 09/2009] = 4,506

This shows that 12,000 infections that were reported on 2/1/2010 were also still present a month later on 3/1/2010. 7,700 infections that were reported from 12/1/2009 were still present on 3/1/2010 and so on.

These are fairly disturbing statistics. What they seem to imply is that once these servers become infected they stay infected. I want to make clear that I am not implying that we have another “McColo situation”. I do not believe that ThePlanet is a bulletproof host nor do I believe they are entirely aware of the duration of some of these infections.

I have tried to contact the abuse department staff of ThePlanet several times over email with these results. I went as far as emailing them directly and asking for their response on some private security mailing lists. If they are reading this now I hope they take us up on our offer to help. As with other infections of this magnitude StopBadware can help out by providing intelligence on where these infections are clustered. We can even provide the lists of hosts provided by our data partners directly to the hosting provider.

I’ve been very interested in applying epidemiology to the world of malware lately. Prevalence is quite simply the number of infected in a given population at a specific time. More specifically it is a ratio of infected over the number of people susceptible. When you look at the data we provide publicly we show you the number of infections for IP addresses and AS blocks. What we don’t show you however is the size of the networks that are infected.
This is something that is likely to change soon. I’m proposing that we start displaying the size of the network by summing up the total number of IP addresses under control of the AS derived from CIDR blocks. This would be fairly trivial for us to do but has some drawbacks. Firstly, CIDR blocks show the size of the network in terms of how many IP addresses are grouped together. It says nothing of how many web servers exist in that range or even how many of the IP addresses are active. This would be similar to saying there are 100,000 houses in zip code 02138 but not saying how many people live in those houses (if any at all). However I’m convinced that knowing the number of IP addresses under the control of an AS block is important.
For instance our page reporting on the top 50 AS block currently shows ThePlanet and Chinanet-Backbone in the number 1 and 2 positions. They have ~16,000 and ~15,000 respectively. However AS4134 (Chinanet) controls 70M IP addresses compared to only 1.5M for ThePlanet. The difference in those two numbers is staggering and it tells me that the number of infections sustained at ThePlanet is abnormally high.

In June we released "a report":http://www.stopbadware.org/home/badwebs with numbers from late May, showing the network blocks containing the largest numbers of badware sites reported by Google. We released updated in "July":http://blogs.stopbadware.org/articles/2008/07/30/updated-infection-stats and "August":http://blog.stopbadware.org/2008/08/25/top-infected-network-blocks-for-mid-august. Here is another update from early October:

# of badware sites	AS block name
35147	CHINANET-BACKBONE No.31,Jin-rong Street
9504	CHINA169-BACKBONE CNCGROUP China169 Backbone
6222	CHINANET-SH-AP China Telecom (Group)
4671	BIZLAND-SD – Endurance International Group, Inc.
4654	CNCNET-CN China Netcom Corp.
3302	THEPLANET-AS – ThePlanet.com Internet Services, Inc.
2460	CRNET_BJ_IDC-CNNIC-AP China Tietong Telecommunication Corporation
1632	SOFTLAYER – SoftLayer Technologies Inc.
1597	PAH-INC – GoDaddy.com, Inc.

Note: A network block owner is not always the owner or operator of the infected servers on that block, and our publication of these data is intended to inform and educate, not to assign blame.

Compared to August, we see that Bizland/Endurance has dropped its number of infected sites by nearly 50%, though it still has several thousand, and Google and NetDirect are no longer on the list. GoDaddy is a newcomer to the list. I just got off the phone with the chief information security officer at GoDaddy, who let me know that they are using the list of infected URLs we provided them to notify customers, offer support in cleaning up the sites, identify the root cause of the infections, and develop proactive strategies for preventing and monitoring site compromises in the future.

StopBadware recently analyzed 49,296 sites which were submitted by trusted third parties to our Badware Website Clearinghouse. We identified five web hosting companies with the largest number of infected sites residing on their servers:

• iPowerWeb, Inc., (10,834)
• Layered Technologies, (2,513)*
• ThePlanet.com Internet Services, Inc, (2,056)
• Internap Network Services, (1,437)
• CHINANET Guangdong province network, (786)

Many of the sites listed in the Clearinghouse as distributors of badware are otherwise innocent sites that have been hacked into by third parties. If a provider hosts a large number of sites that distribute badware, it’s possible that the provider has unaddressed security vulnerabilities that increase the likelihood of the sites the provider hosts being hacked.

iPowerWeb, one of the world’s largest hosting providers, hosts a startlingly high number of sites in the Clearinghouse. iPowerWeb’s homepage claims that the provider hosts over 700,000 sites; at 10,834, more than 1.5 percent of those can infect internet users with badware.

StopBadware spoke with some owners of websites in our Clearinghouse that are hosted by iPowerWeb. To iPowerWeb’s credit, many of their customers report that iPowerWeb personnel quickly located and removed the badware or badware-distributing code from their sites. Some customers complained, however, was that iPowerWeb support personnel were unable to provide details about how the websites were compromised.

StopBadware encourages all web hosting providers to work proactively to stem the spread of badware on the internet. StopBadware co-director John Palfrey says, “Web hackers and badware distributors are constantly finding new ways to work around the safeguards that are put in place to protect consumers. Web hosting providers must do their part to stay ahead of the curve and help keep the websites they host safe from malicious attacks.”

You can read our full press release here. If you’d like to comment on this information, or share a story about your own experiences with a web hosting provider to address a compromised site, please visit our discussion group.

* NOTE: Layered Technologies has informed us that it provides a style of web hosting known as self-managed hosting, in which its customers have full control over and responsibility for server management.