Updated Blacklists
I’m pretty attentive when it comes to securing my server and I can be exceptionally pedantic when deciding what constitutes bad behaviour or misuse of my webserver. Typically this would include crawlers that ignore robots.txt and script kiddies looking for certain versions of software etc.
To that end I tend to use a combination of blacklisted networks which are blocked on the webserver using shorewall and user agent matching which is performed on my reverse proxy.
For the past two days I have had repeated attempted crawls from Chinese netspace. All looking for the same vulnerabilities and almost all using libwww-perl as their useragent, an example of which is below:
61.151.239.75 – - [28/Jan/2009:18:08:38 +0000] “GET http://blog.sweetnam.eu:80//blogtest/xmlsrv/xmlrpc.php HTTP/1.1″ 403 2371 “-” “libwww-perl/5.803″ TCP_DENIED:NONE
The crawler above appears as being from CHINANET Shanghai province network but I have had literally thousands of scan attempts from many different Chinese addresses. So once more I have decided to completely block all of China from accessing my webserver in addition to the other hosts that I block as well.
The link below contains a list of IP addresses that my firewall rejects requests from. The first 763 lines are from China alone.
Cheers for the list, I was getting same hits too.
You’re welcome Matiki,
Some might think it unfair to block an entire country but unfortunately that is where most of my crawlers come from (And the Ukraine).
But at the end of the day it is my site and my money being used for wasted bandwidth.
Anyway, Glad I could help.