tech.sweetnam.eu

Recently I decided to go into the hosting business. Not in a big way mind you. Just to cover the costs of hosting primarily with perhaps enough left over for a night on the town once a month or to pay the phone bill.

As it happens I was lucky enough in that I had two clients ready to go with over 100 sites each. Starting up a business without having to look for clients is an absolute luxury especially in the middle of one of the worst recessions this country has ever seen. After making all the agreements with my future clients it was time to come up with a concrete migration plan.

Before a single site could be migrated several things needed to be taken into consideration:

• Hardware
• Software (Operating System and Control Panel)
• Support/Knowlegebase/Helpdesk
• Bandwidth requirements
• Data import/migration/compatibility

The hardware was the easiest. I had a spare Dell poweredge 1850 hanging around here at home doing nothing with a not too inconsiderate 6GB of RAM.  The operating system was going to be a simple matter too. It was going to be Linux or nothing. However which Linux distribution to choose?

Choosing the Linux distrubution was going to be directly related to my choice of control panel software. I wrote here sometime back about Virtualmin and I decided that it would be absolutely perfect for my control panel requirements. Having decided on the control panel software it was now up to me to choose between CentOS or Ubutu 8.04LTS as the host OS. In the end out of familiarity I opted for Ubuntu.

One of the key reasons for choosing Virtualmin was its ability to import backups from cPanel. As I would be migrating almost 200 sites from a cPanel server in the U.S. the ability to seamlessly migrate would be an absolute bonus. Another important reason was that the control panel interface itself is very easy to use. Considering that my two future clients were coming from years of using cPanel I was confident that they would easily find their way around Virtualmin.

In February I installed the server into the datacentre and pulled across a couple of cPanel backups to test the import functionality. Of primary concern to me was the existing server was running CentOS and using Exim as its MTA. The new server was the already mentioned Ubuntu and I had decided on Postfix as the MTA. In addition the home folders of the existing server were split between two disks mounted as /home and /home2

So I pulled across two backups, one with /home as its location on the old server and the other with /home2, used the import function and in about 5 minutes both sites had been migrated flawlessly.

I couldn’t have been that lucky I thought to myself. Normally something goes awry especially when there are significant differences in software as well as software versions. I poked around all the config files and was astonished to see that everything looked as it should be.

Over the next week I made out a schedule for migration and before long all sites were up and running. My clients were happy as too were my clients clients. The only issues that cropped up were minor and were as a result of differences in the way Virtualmin handles user accounts compared to cPanel.

So that was over a month ago and with everyone happy I can reflect on what was accomplished. Most notably was there anything I could have done differently or more efficiently.

As it happens yes, there was something I could have done had I thought about it.

The hardware of the server is absolutely overkill for what it is doing. 6GB of RAM is way too much. Over the past month the most I have ever seen in use was just short of 800MB however having lots of free RAM is not a bad thing as Linux likes to use lots of it for cache.

The load on the old server was constantly around the 1.00 to 2.50 mark, the new server with it’s dual core Xeons is barely even breaking a sweat with load averages between 0.03 and 0.12.

What I should have done, and it’s obvious now, is that I should have installed a hypervisor like VMware ESX server and paritioned the physical machine in to three virtual machines. One for each of my two clients and the third for myself.

In any event I have another Poweredge 1850 and a pair of 1750′s that I intend on installing into the datacentre in the not too distant future so now it is time to start planning for that.

To finish off here’s a screenshot:

Way back towards the end of August I inherited some fairly half decent equipment. Among them were two Symantec  5420 Security Appliances. I took them despite not knowing what it was that I was going to do with them. After a while I simply forgot about them until I decided yesterday to do something.

As the licenses had expired they weren’t much use as a security appliance so first things first was to take the cover off and have a gander inside. As I suspected an €8,000 from Symantec would of course be made from the cheapest parts available. The motherboard was from a company I had never heard of in all my years called iWill. Any references on the web point to them no longer existing. The CPU is a 2Ghz Celeron and it had 512MB RAM and a 40GB Maxtor hard disk. Connectivity wise there are 6 10/100Mbit Intel ethernet ports, two USB and one RS-232 ports. Inside on a riser board is a PCI slot however fitting a normal PCI card such as a display adapter s out of the question as the IDE cable would block even the shortest ones and besides, VGA cards are not supported! In the second IDE slot is a 16MB SSD.

So what was I going to do to get a new OS up and running on it? I came across a forum post from a guy who managed to get pfsense (FreeBSD) installed on it but unfortunately his instructions were a bit lacking. Out of curiosity I hooked a console cable up to the RS-232 port and opened a terminal where I was quite pleased to be greeted by a login prompt. I logged in using the username admin with the password I got from the LED panel on the front and lo and behold I was in a root shell. I guessed that Symantec’s software probably ran on one of the BSD’s or Linux from experience with Nokia Checkpoint which ran NetBSD I think.

Anyway I guessed correctly, the 5420 was running RedHat Linux 7.1. From the console I was able to garner a bit more about the hardware. In addition to the basic stuff I mentioned earlier there is a Broadcom Cryptographic Accelerator CPU on board. But back to my immediate problem. How to get another OS on it. After pondering for a few moments I decided to try and see if taking the disk out and sticking it into a spare PC and setting it up from there would work. I dusted off my trusty Optiplex Gx1 circa 1999 and proceeded to install Ubuntu 9.10 on the disk.

Before I stuck the disk back in the 5420 I made sure that I could access a shell via console cable. Thankfully there is some excellent documentation on the Ubuntu help site and in no time I was able to access the Dell via console cable and have access to the Grub boot menu on boot up as well.

Back into the 5420 did the disk go, console cable hooked up, terminal open and time to power up. To my absolute delight there was the Grub boot menu. I hit enter and up it loaded. Perfect! Except for one thing. I had no network access. Half way through diagnosing this the whole thing just froze on me so I rebooted. Not long after rebooting it froze again. It was then I remembered that the guy who installed pfsense on his one mentioned problems with ACPI.

Unfortunately as it would freeze up so quickly I was unable to fix it from a terminal so I had no other choice but to take the disk out and shove it back into the Dell again. Ubuntu 9.10 has a lot of things moved about and the new version of Grub had me a bit confused for a while but finally I got ACPI disabled and everything is working perfectly.

Being headless they aren’t much good for anything other than possibly their original intention as a security appliance. However I decided to set up one of them as a dedicated monitoring server running ntop and Munin. For that it is absolutely perfect!

It’s been absolutely ages since I last wrote here and I had been meaning to do so for some time. Since my last entry I have been tinkering away with my setup as I do on a regular basis and plenty of things have changed here on the back end. I suppose I will begin with my my connection and move up the chain from there.

First up is a new router. I picked up a nice Cisco 837 for a very reasonable price. I had a bit of fiddling to get it up and running with eircom broadband but I got there in the end and I have documented the procedure over on my wiki.

Next up is my reverse proxy. It is still running on the same hardware but I decided to move from Linux over to OpenBSD. My primary reason for this is that I was getting sick and tired of Linux iptables. In addition I concluded that Squids performance wasn’t optimal either. A move to OpenBSD seems to have resolved both those issues for me. PF on BSD is very configurable and easy to understand as well. Needless to say I have documented my experiences with PF over on my wiki also.

I’ve also gotten my hands on some new hardware. Well new hardware to me that is. I picked up three Sun Netra T1 servers, a Dell PowerEdge 1750 and two Symantec 5420 Firewall appliances.

So far I have been busy setting up the three Netras. They don’t have CD-ROM drives or display adapters so I had to install them over the network which was a process that was actually relatively simple. However it was only simple in that I already have a couple of other Sun machines here one of which I was able to use as an install server. Of course I have this process documented on my wiki also.

I haven’t decided what I will use the Netras for yet. One of them seems to have CPU problems and has been cannabilised to increase the specs. of the other two. They have 500Mhz UltraSparc IIe CPUs so by todays standards are not at all that powerful but they do draw very little power, certainly less than a Pentium IV machine so I may set them up as file servers. Initially I was hoping to install OpenBSD on one of them to use as my reverse proxy but unfortunately an OpenBSD network install does not seem to be as straight forward as Solaris.

The Poweredge 1750 I had hoped to set up as a webserver running either IIS or Apache. Unfortunately it only has 512MB of RAM so until I get some more for it that rules out Server 2008. However it is incredibly loud and that rules out keeping it running in my computer room. I am currently planning on setting up a dedicated computer room as it were in another part of the house so check back here for update on that.

Having being obsessed with computers and networks for many years I am always interested to find out the infrastructure behind some of the more popular sites out there. Quite often it is possible to glean bits of information here and there and occasionally an error may occur that offers a glimpse as to what is happening in the back end.

Today it is the turn of the Irish Independent. I got the error pictured here while browsing through their site. What is interesting about it is the domain name; externalcontent.independent.ie. As we can see it is an served by an Apache web server running on a Red Hat machine. However if you look at the error more closely it is a bit more telling.

The server in question (externalcontent.independent.ie) attempted to serve up an ad or content related to  loadzajobs.ie but was unable to contact the back end server. So this tells me that externalcontent.independent.ie is configured as a reverse proxy server and according to Netcraft is located in Ireland.

The primary domain; www.independent.ie, is hosted in the Netherlands and runs Apache Coyote, again according to Netcraft. Apache Coyote is a connector for Apache Tomcat. I find it curious that the main www site is located in the Netherlands but I suspect that it might have something to do with being connected to the Amsterdam Internet Exchange which is largest Internet Exchange in the world.

It does make me wonder why they chose AMS-IX given that here in Ireland we have INEX.

So what about the other national daily online newspapers?

The Irish Times is hosted on Linux and Apache and hosted in Dublin and The Irish Examiner is hosted on Windows Server 2003 and Microsoft IIS/6.

Although there are Irish editions of the Sun, The Star, The Daily Mail and The Mirror, none of them have specific Irish orientated sites but I will include them here nonetheless.

The Sun claims to be hosted on Linux and Apache however they use Akamai for content delivery so this could be inaccurate.

The Daily Star claims to be hosted on an unknown Unix and Apache and the Daily Mirror is hosted on RedHat Linux and Apache.

Finally the Daily Mail, like the Sun also uses Akamai for content delivery and claims to be running Linux and Apache.

It’s clear that Apache and Linux are the front runners.

I’m pretty attentive when it comes to securing my server and I can be exceptionally pedantic when deciding what constitutes bad behaviour or misuse of my webserver. Typically this would include crawlers that ignore robots.txt and script kiddies looking for certain versions of software etc.

To that end I tend to use a combination of blacklisted networks which are blocked on the webserver using shorewall and user agent matching which is performed on my reverse proxy.

For the past two days I have had repeated attempted crawls from Chinese netspace. All looking for the same vulnerabilities and almost all using libwww-perl as their useragent, an example of which is below:

61.151.239.75 – - [28/Jan/2009:18:08:38 +0000] “GET http://blog.sweetnam.eu:80//blogtest/xmlsrv/xmlrpc.php HTTP/1.1″ 403 2371 “-” “libwww-perl/5.803″ TCP_DENIED:NONE

The crawler above appears as being from CHINANET Shanghai province network but I have had literally thousands of scan attempts from many different Chinese addresses. So once more I have decided to completely block all of China from accessing my webserver in addition to the other hosts that I block as well.

The link below contains a list of IP addresses that my firewall rejects requests from. The first 763 lines are from China alone.

http://tech.sweetnam.eu/blacklists/blacklist.txt

When I originally decided to host from home a few years back I had to do more than just decide to fire up a webserver and NAT from my router to it. Choosing an operating system and web server software and application server was another consideration. Did I want to run Linux? Solaris? Microsoft IIS? and did I want to go with PHP or ASP? What was I going to use as a CMS for my primary site and what blog software would suit my requirements?

Ultimately I decided on OpenBSD with Apache and PHP and Windows + IIS. Of course I knew that over time I would be constantly changing this and I needed a way to effictively manage the domains themselves and the sites and database dumps. Obviously I needed a control panel of some sorts and after a bit of investigation I settled on Virtualmin.

Virtualmin is created by the same people who created the excellent and utterly essential Webmin control panel and as such the Virtualmin module integrates nicely into Webmin. Oh, and it’s free!

Originally this site was hosted on Windows Server and IIS and powered by BlogEngine.NET but I found it a bit of a pain to handle two different types of blogging software so I migrated this blog over to WordPress and host it with my other sites. Over time my other server changed several times from OpenBSD to Ubuntu, then Solaris 10 running on a Sun Workstation, back to Ubuntu and currently onto the Debian machine where they currently reside. With Virtualmin moving platforms was a simple matter of restoring from the backups that I took from the machine that was to be replaced.

Of course with it now being relatively trivial to change servers I find myself experimenting a bit more and I’m currently fighting the urge to move everything over to a FreeBSD server but for the moment I’m successfully resisting that urge!

I tend to have multiple ssh sessions open at once to my multiple *nix machines. However despite all the best attempts to keep some order all the windows tend to clutter up the desktop. Gnome terminal goes some way to easing that pain by supporting tabs but it would be even nicer if I could have multiple consoles all in a single window.

After a bit of digging I came across Gnome Terminator.

It’s a rather nice simple terminal console app but where it wins for me is that if you right click inside the terminal you can split the terminal horizontally or vertically or any combination of both. If you see my screenshot below it will give you a good idea. And yes almost all my machines are names after characters in Stargate!

Chances are that there are already pre-built binaries available for your distribution. At least there is for my Ubuntu workstation so it was a simple

sudo apt-get install terminator

It’s certainly worth checking out if like me you like to have lots of console windows open.

Terminator in action

I had a hard drive failure on my main workstation recently and after installing the new one I decided to partition it with Windows XP and Ubuntu 8.04. That was about six weeks ago and I have only booted into Windows once.

I would consider myself a pretty advanced Linux/Unix user. My first encounter with Linux was in 1995 with LinuxFT and not long after that with an early Slackware release. As the years have rolled by since then I have setup literally hundreds of Linux servers and scores of Solaris and HP-UX servers but Linux has never impressed me on the desktop.

Through one job I had eight years ago I had to use RedHat 6.1 as a desktop OS and I absolutely hated it. The look and feel was terrible. The early GTK widgets were hideous. Gnome was at version 1.4 and it was almost completely unusable. Around the same time I installed SuSE 6.0 on a spare PC at home and while it used KDE by default, it wasn’t much better. Installing fonts was a nightmare and the only decent browser available then was Netscape Navigator which used the Motif Widget set and was a nightmarisly ugly looking beast with which to browse the web.

How things have changed in eight years! A quick roll call of my computer room here at home comes up with the following:

1 HP 9000 running HP-UX
1 Reverse Proxy server running Ubuntu 6.06 LTS
1 Web Server running Debian Lenny 64bit
2 Sun Blade 2000 workstations running Solaris 10
1 Apple MacBook running OS X 10.5 Leopard
1 Dell Poweredge running Ubuntu 6.06 LTS
1 Main workstation running Ubuntu 8.04 LTS (Dual booting with Windows XP)
1 Firewall running Smoothwall Linux

And finally:

1 File server running Windows Server 2003.

I’m now down running just one single Windows machine from a maximum of about 8 over the years. And unlike before I don’t think I will be turning back. Most of the applications that I used on Windows were open source to begin with and naturally they have Linux versions. Firefox and Thunderbird being the most obvious two. WINE has matured to such an extent that when I’m no longer comfortable using The Gimp for certain tasks, Photoshop CS2 works under WINE like a charm.

However if there is a problem with Linux it is that there is probably too much choice. Particularly when it comes to your default desktop. I’ve finally decided on Gnome mainly because I couldn’t get Compiz working properly with XFCE. I had a look at KDE4 and I will probably have to wrte a seperate post about it. Suffice to say for the moment I don’t like the look of it one little bit.

So after six weeks I finally have my desktop looking the way I want it to. Gone is the Ubuntu Brown default theme replaced instead with one I liked from art.gnome.org . I’m still trying to find window decorations that I like but for the moment the default Ubuntu one is tolerable.

I still do think however that Linux is still no where even near ready for use as a mainstram desktop OS for the average user. But for the moment I’m finally impressed enough that I can eschew Windows at long last.

Want to try something new? Sun’s OpenSolaris is now available for download. OpenSolaris is as the name suggests an open source edition of Sun’s flagship Solaris operating system. However in actuality it is quite different. It seems to be aimed squarely to compete as a viable alternative to the more popular Linux distrubutions. To that end it comes on a live CD like Ubuntu.

Among the notable features is the use of ZFS as the default filesystem and also includes the Xen hypervisor for builtin virtualisation support. It also included a package manager with software available from repositories like Ubuntu. I imagine that Sun intend for technology that is included in OpenSolaris to make it upstream into later versions of the venerable Solaris OS.

At the moment I’m downloading the CD and I’m going to install it on a virtual machine for a look see.

No doubt you have guessed that I’m something of a Sun fan and at times I think to myself that this blog is turning very Sun orientated. This is not the case however as I recently upgraded my mail server to Ubuntu 8.04 from 6.06 and I have no intentions of changing that anytime soon. However I recently moved my main site and personal blog over to one of my Sun Blade 2000 workstations. And of course this blog is still powered by Windows and IIS.

Diversity? I’ve heard of it.

Here is an interesting nugget of information for you. The project leader for OpenSolaris is non other than Ian Murdock who was one of the original founders of Debian Linux on which Ubuntu is based.

I’ve had a busy few days patching upgrading and moving sites around.

The two biggest changes were upgrading my mail server from Ubuntu 6.06 to 8.04 which went very smooth indeed and moving all sites hosted on my Fedora powered Dell Poweredge over to Solaris 10 running on one of my Sun machines.

As well as the above I also moved my reverse proxy from an elderly Pentium III 550 to a much more modern and powerful P4 3Ghz machine. I’ve also moved my outgoing proxy onto the second of my Sun Machines which is now running the latest version of Squid which at the time of writing this is 3.0 STABLE4. All my windows machines have also been patched and updated and my last remaining XP machine is now eagerly awaiting the release of service pack 3 which is being released tomorrow. Come to think of it I’ve just realised that my XP machine has been running the preview release of SP3 since it was released some time back (v.3264). No doubt I’m going to have a nightmare task ahead in installing the proper release.

But getting back to the Dell Poweredge and why I moved everything over to one of the Suns. I have a task in mind for the Poweredge but this task involves Windows Server. I actually have a licensed version of Windows Server 2003 for it so I have to say goodbye to Fedora