tech.sweetnam.eu

It’s been absolutely ages since I last wrote here and I had been meaning to do so for some time. Since my last entry I have been tinkering away with my setup as I do on a regular basis and plenty of things have changed here on the back end. I suppose I will begin with my my connection and move up the chain from there.

First up is a new router. I picked up a nice Cisco 837 for a very reasonable price. I had a bit of fiddling to get it up and running with eircom broadband but I got there in the end and I have documented the procedure over on my wiki.

Next up is my reverse proxy. It is still running on the same hardware but I decided to move from Linux over to OpenBSD. My primary reason for this is that I was getting sick and tired of Linux iptables. In addition I concluded that Squids performance wasn’t optimal either. A move to OpenBSD seems to have resolved both those issues for me. PF on BSD is very configurable and easy to understand as well. Needless to say I have documented my experiences with PF over on my wiki also.

I’ve also gotten my hands on some new hardware. Well new hardware to me that is. I picked up three Sun Netra T1 servers, a Dell PowerEdge 1750 and two Symantec 5420 Firewall appliances.

So far I have been busy setting up the three Netras. They don’t have CD-ROM drives or display adapters so I had to install them over the network which was a process that was actually relatively simple. However it was only simple in that I already have a couple of other Sun machines here one of which I was able to use as an install server. Of course I have this process documented on my wiki also.

I haven’t decided what I will use the Netras for yet. One of them seems to have CPU problems and has been cannabilised to increase the specs. of the other two. They have 500Mhz UltraSparc IIe CPUs so by todays standards are not at all that powerful but they do draw very little power, certainly less than a Pentium IV machine so I may set them up as file servers. Initially I was hoping to install OpenBSD on one of them to use as my reverse proxy but unfortunately an OpenBSD network install does not seem to be as straight forward as Solaris.

The Poweredge 1750 I had hoped to set up as a webserver running either IIS or Apache. Unfortunately it only has 512MB of RAM so until I get some more for it that rules out Server 2008. However it is incredibly loud and that rules out keeping it running in my computer room. I am currently planning on setting up a dedicated computer room as it were in another part of the house so check back here for update on that.

I’m pretty attentive when it comes to securing my server and I can be exceptionally pedantic when deciding what constitutes bad behaviour or misuse of my webserver. Typically this would include crawlers that ignore robots.txt and script kiddies looking for certain versions of software etc.

To that end I tend to use a combination of blacklisted networks which are blocked on the webserver using shorewall and user agent matching which is performed on my reverse proxy.

For the past two days I have had repeated attempted crawls from Chinese netspace. All looking for the same vulnerabilities and almost all using libwww-perl as their useragent, an example of which is below:

61.151.239.75 – - [28/Jan/2009:18:08:38 +0000] “GET http://blog.sweetnam.eu:80//blogtest/xmlsrv/xmlrpc.php HTTP/1.1″ 403 2371 “-” “libwww-perl/5.803″ TCP_DENIED:NONE

The crawler above appears as being from CHINANET Shanghai province network but I have had literally thousands of scan attempts from many different Chinese addresses. So once more I have decided to completely block all of China from accessing my webserver in addition to the other hosts that I block as well.

The link below contains a list of IP addresses that my firewall rejects requests from. The first 763 lines are from China alone.

http://tech.sweetnam.eu/blacklists/blacklist.txt

I’ve written before about how I use my reverse proxy to block various bad bots and crawlers. At this stage I am blocking so much stuff that it would be way to much to post here. So if you are interested here are two links for you. The first is a copy of my squid regex file I use as per my tutorial and the second is a list of IP address and IP block that I have blocked on the proxy using IP tables.

Blockedbots.txt
Blockedip.txt

Update – I have a more complete tutorial on how to block bots with Squid over on my wiki which you can view here.

I’ve written before about my reverse proxy and how it allows me to accelerate content delivery and also to allow me to run multiple webservers unsing a single IP address. However it is capable of so much more.

Squid uses access control lists (acl’s) to govern who can do what with the proxy server. For example you can set acls to only allow certain computers to access the internet or indeed access the internet via the cache at certain times or hours. There are a myriad of different options that you could configure but one in particular struck me as being exceptionally useful. That is that you can use acls to block certain useragents.

In a conventional scenario you would use .htaccess on the server to block access to various bad bots. If you were the administrator of several or maybe even a few dozen sites then it becomes a chore to ensure that the bot and nefarious useragents in all the .htaccess files are kept up to date. However as in my case as all traffic is passing through the reverse proxy it becomes trivial to deny access to those bots and useragents as all you have to do is create a single acl and it will apply to all sites that the proxy is fronting for.

Setting it up couldn’t be easier.

In my case my squid.conf is almost identical to the one used on my reverse proxy tutorial. One of the key things to consider in adding an acl to block certain useragents is that the new acl that we will be creating needs to be read by squid on startup before all the others.

First up we need to define our acl. So as per my tutorial I need to add this acl which I will be calling ‘badbrowsers’ just above the first ‘cache_peer’ entry in squid.conf. I will be storing all the bad bot entries in a seperate text file to avoid a messy squid.conf. In order to get squid to reference a seperate file, the location for the file musr be enclosed in quotes. So now we define our acl exactly as follows:

acl badbrowsers browser “/etc/squid/badbrowsers.conf”

Now the acl has been defined we must decide on an action that will occur when our new acl is triggered and for this we need to scroll down through our squid.conf and in a new line just above the http_access for our proxied sites add a new line to deny http access for out acl as follows:

http_access deny badbrowsers

That’s all the configuration needed for our squid.conf so save your changes and now we will create and edit the file that we have defined that will contain our bad bots and useragents.

When defining our acl the configuration file that I have chosen will be located in /etc/squid. So change to this directory and using your favourite editor create a file called badbrowsers.conf. On each line in this file we can add our banned useragents using regular expressions. I’ve noticed lately that most of the comment spam that I have been receiving lately has been coming from a useragent calling itself “Jakarta Commons-HttpClient/3.0.1″. To banish this useragent add a line to your badbrowsers.conf file with the following:

^Jakarta

That’s it. That’s all you need. Once the first word is matched in the useragent string you don’t need anything else. You can elaborate on this if you like to encompass whatever you like using regular expressions.

Once you are happy with your configuration save your changes and restart squid and no more bad bots.

For the HowTo on my wiki please click here.

This is a repost from my personal wiki originally published on 18th August 2007

It seems that I can never leave well enough alone as far as my home setup is concerned. At one stage I had an excrutiatingly complicated
mail setup. Somewhile back I moved to rectify that and now I just have
the one mail server which is also serving mail for a few other domains.

My webserver was just running on one machine and is also serving up a few other domains for friends as well.

But I decided it was too simple.

One thing that was bugging me was how to make the most out of having
one static IP address. Virtual hosts are an option on Apache which fits
the bill nicely if all you want to do is serve PHP and static HTML
sites. But if you want to extend of further and run a J2EE app or an
ASPX site you pretty much hit a brick wall.

This is where reverse proxies come in. You might know that a proxy
server serves requests from multiple clients to multiple servers.
Acting as a kind of gateway. They also cache frequently accessed files
so they can also help to reduce bandwidth. Especially in a situation
where many users are sharing a single internet connection. A reverse
proxy as the name suggests does the reverse.

Essentially it allows multiple connections from the internet and
depending on certain criteria routes the requests to the desired
computers in your local area network. So I set one up today. In fact
the page you are reading now has passed through the reverse proxy.

There are many solutions available to implement a reverse proxy. The
Apache webserver would be one of the better know ones along with the Squid Cache
proxy server. Both of these are open source which means that they are
free so I opted for Squid as it is a lot more configurable.

So after an afternoon of compiling and configuring I managed to get
it up and running. Rather than bore you with the details now, I intend
to do a write up on my wiki in the near future. Not least of all so
that I can remember what I did if anything goes wrong.

Anyway, as already mentioned you are looking at this page served up
to you by IIS running on Windows. If you follow the links to the howto you find yourself viewing a page running Apache on Linux.

About the only ‘gotcha’ I noticed is that all my logfiles show the
requests as coming from the proxy rather than the actual computer that
made the request. I can use the actual squid logfiles from now on but
I’m going to have to work on implementing a solution for that soon.

Update – I have added a howto to my wiki. You can access the howto here.