When you maintain several servers each running several virtual machines and have anything else at all to do, it is impossible to keep your eyes on them 100% of the time. There are a handful of software packages out there that can do this for you, but most are either too bulky, too complicated, or lack the features that you want. Scott Pinkston referred me to one the other day called Monit, which is the first one of these programs that I actually liked. Most of the others just have too much. This one is short, and to the point. Monit installs very quickly, and runs in the background as a daemon. One very simple config file holds the configuration data for the services you want to monitor, and if you so choose, you can have a secure webpage display stats on your services. It can also be configured to email you (or call the beeper, etc...) if certain events occur, and best of all, is super lightweight.

As I wrote about last time, Pleth's move from VMWare Server to VMWare ESXi has been very successful thus far, but in the process we've discovered a couple of "neat tricks" and have proven to ourselves that the technology choices we made a few years back were indeed the right ones.

When you copy a .vmdk (vmware disk image) over from a VMWare Server machine, you have to convert it over to ESXi format. This process makes the resulting disk image the whole size that you've allocated. This isn't necessarily a bad thing, but if you had it set to thin provisioning in VMWare Server your disk usage just went up. WAY up.

As I posted last time, we decided to move over to ESXi and so far, its been pretty smooth. ThePlanet installed ESXi 3.5 on our servers, which I quickly upgraded into 4.0. When you install the vSphere Client there is an option to install the host update utility. Run it, point it to the zip file you've download from VMWare's website, and wait a bit. It works like a charm (put the machine into maintenance mode first!!). Since the servers had no clients running on them, I did it during the day (which let me sleep last night! lol!!) I've been copying over the VM's from our VMWare Server machine with good ol' scp and using the vmkfstools command on the ESXi box to convert them into ESXi format. Takes about 30-40 min per server for the whole process, which isn't exactly quick, but we're moving low-traffic boxes in very off hours. I moved the server that this website runs on during lunch today....;o)

I'm teaching at UACCB this summer, and my Microeconomics class is taking their fist test. There's not a lot of smiles in the room....(which means my work here is done...lol). Anyway, spent the morning reading about tuningVMware ESXi, which was just released in version 4.0, and supports hardware that we can get in our private shelf at ThePlanet. We ordered 2 new servers this week, both dual-processor Quad Core Intel Xeon powered Dell Poweredge models with 16gb ram and a ton of disk space. This should make things a bit faster than our current configuration. We were going to migrate everything to VMWare Server but with the release of 4.0 and ThePlanet supporting ESXi in their datacenters, this move makes much more sense.

I posted a few weeks back that Pleth had transitioned some of their equipment over to VMware Server and for the most part it's been a very smooth process. But, as of late we've ran into some slowdowns, especially on the VPS with Plesk (which happens to host several of our websites). After doing a bunch of research and spending many a late hour digging through tons of mpstat and other sysutils data I think I found the culprit(s).

VMware Server, unlike the ESX/ESXi products, does not run in a Type 1 Hypervisor. This means that the underlying OS (in our case Red Hat Enterprise Linux was tuned out of the box for a general all-purpose server. This configuration isn't always optimal for a Type 2 Hypervisor. It works just fine as long as things are "normal," but as the new VMware server got a larger load (in terms of I/O and CPU) performance went downhill.

Since I'm back, I've got a few days worth of log files to dig through. A couple of years ago an old legacy PHP script Pleth was running wasn't very secure, but was critical to the operations of a particular customer. It got hacked (well, they used it to upload a C99Shell) a couple of times before the vendor released an update. Scouring the internet for a solution, I learned of Mod Mod Security, an application firewall of sorts. It runs as a module in your Apache configuration and uses a set of user-configurable rules files to detect and prevent a number of attacks against a website. The rules list has a huge community backing, and people have written rules for about every vulnerability out there. Open Source is good no? Anyway, as I was digging through those files today it kinda shocked me to see just how much stuff mod_sec blocked. The internet is a dangerous place.....

I just got off a phone call with Craig at First Community Bank where I just explained how to use vi to to edit a crontab. I am such a geek......

Those that know me know I do a lot of server work for Pleth, LLC and that most Fridays I can be found there. We've been in the process of moving from a Parallels Virtuozzo environment to a VMWare environment. So far, the process has went well. We have a private rack in our datacenter at ThePlanet and just installed a nice new system with quad cores, tons of ram and storage. It's running VMWare Server 2 on Red Hat Linux 5 with Centos 5.3 VM's. One of the VPS's on this hardware node has a 100 domain Plesk 9.2 installed on it, and we've been moving some of the domains from the old Virtuozzo/Plesk containers onto this machine. VMWare has made it very easy to be portable from one hardware platform to another because of the OS independent nature of its Virtualization environment.