A month or two ago, we bought two new servers for the toolserver. One of these (nightshade) was to be the new login server (replacing hemlock, which would remain as the web server), and the other (willow) the new stable server (replacing the current stable, which is a virtual server on vandale). These are Dell PowerEdge 1950 IIIs, each with 2 quad-core 2GHz Xeons (E5405), 8GB RAM and 2 146GB SAS disks. Compared to the current login server, a dual Opteron, this is quite an improvement.

Unfortunately, when they arrived, we realised we’d forgotten to buy DRAC cards, so we couldn’t use them. (Dell Remote Access Controller is a management card that allows remote power control and serial console access to servers.) Last week, we finally got two DRAC cards installed, so I began setting up the new servers. Mostly, this was pretty simple. A few minor problems:

• The Debian Linux installer doesn’t support the Broadcom NetExtreme II network card, because it requires a so-called “non-free” firmware to be loaded. I fixed this by creating a custom initrd containing the firmware file, after which installation was relatively simple.
• Solaris does support the NetExtreme II, but it doesn’t support the PERC6/i RAID controller. Again, the fix for this was building a custom install miniroot containing the LSI driver for this card (mega_sas).
• Serial console redirection on these servers defaults to “continue after POST”, which has an unfortunate interaction when GRUB is configured to use the serial console; the server appears to hang after POST. This is easy to change, but annoying if you don’t notice it.

Since we now have two user servers, the next issue was sharing user account information between them. The simplest way to do this is just sycning the account data (/etc/passwd, /etc/shadow and /etc/group) between the two servers. There are a few problems with this, though: you have to wait until the next sync for updates to take effect on both servers; and any differences between the two on each server (for example, Debian likes to add new users during package installations) will be lost.

So, the two common options for shared account information are NIS and LDAP. NIS is an older system that provides basic key/value maps (e.g. looking up the username in the ‘passwd’ map, and getting the corresponding entry from /etc/passwd). On the other hand, LDAP provides a database of objects, each of which can have any number of user-defined attributes. The advantage of LDAP is that each user object can have additional information beyond POSIX account information (for example, email accounts). I decided to use LDAP for the toolserver, mainly for this reason.

The LDAP server I used was Sun Directory Server Enterprise Edition. Despite the name, this is a fairly lightweight LDAP server, with good documentation. In the past I’ve used OpenLDAP, but I found it quite fiddly to configure, and not so well documented. The Linux version of DSEE is only supported on RedHat and SUSE systems, but it installed and ran fine on Debian.

Importing the existing account data to LDAP using the PADL LDAP MigrationTools was quite straightforward. For some reason, Debian has two different LDAP NSS modules; libnss-ldap and libnss-ldapd. I chose the latter, which uses a daemon (nslcd) to proxy requests between user applications and the LDAP server. After configuring NSS and PAM to use LDAP, everything seems to be working.

The new stable server was simpler to set up, as it’s replacing the existing server rather than augmenting it. Installing Solaris and the web server was straightforward; all that was left was copying a few configuration files from the old stable server.

One thing I did differently was using pkgbuild for installing software packages, rather than compiling by hand. pkgbuild builds Solaris packages from RPM spec files; as specs contain all the information required to build a package in a single file, it’s easy to create repositories of spec files which can be built using pkgbuild. This is much easier to manage than the old stable server, where things were simply built and installed without using packages. And, as there’s already a large existing repository of such specs (spec-files-extra) to work from, it’s easy to add new software.

These new servers should provide enough capacity for the toolserver to last a long time, as well as improving both interactive performance and web serving. (But look out for a future post about switchboard, a fast replacement for suphp using FastCGI — if I ever get around to finishing it.)

A lot of Toolserver users run a MediaWiki bot framework called pywikipediabot to perform maintenance tasks on Wikipedia, such as fixing spelling errors or double redirects, and adding interwiki links to articles.

Unfortunately, some of these bots are rather excessive in their memory use. In the past, it was usual to see a few interwiki bots using 500MB+ each. Recently, support for caching to disk (instead of using RAM) was added to pywikipediabot. Hopefully, this will help reduce the memory use of these bots. Next on the list: a Java program called “Linky”, which is apparently some kind of IRC bot. There are 9 copies of Linky running on the toolserver, using 100MB each—that’s 900MB, or nearly ⅛th of the system’s total memory—dedicated to just that.

The problem with running a shared server is that users often don’t realise how many resources they’re using. If you’re running a bot at home, using 100MB RAM is probably fine. But when you have to share available resources with other people, you need to start looking more closely at your resource use, and perhaps spend some time reducing it. (If you think about it for a moment, 100MB just for an IRC bot is a little excessive, isn’t it?)

The main technical restriction for memory use on the toolserver is a homemade daemon called slayerd, which starts killing users’ processses when they use more than the limit (currently set at 1GB). This is mainly intended to stop runaway processes than enforce policy—usually an administrator will start killing processes long before 1GB, if they’re noticed. The only way to really fix the problem is to educate users about the problem. Of course, the problem then is treading the line between gentle reminders, and whining ;-)

Sunday morning (5-7AM UTC) is the schedule maintenance window for the Toolserver. During this time, we apply patches (software updates), and do any other work that might result in downtime. Now, patching a system is an inherently dangerous process: you’re changing the software running on the system, and there’s always a chance of new bugs being introduced. Of course, patches go through QA testing before being released, but sometimes problems still slip through. And, since many patches are large, and have to be installed in single user mode (where no services are running), even a harmless patch means downtime during installation.

Live Upgrade

At the Toolserver, we mitigate patching problems with a process known as Live Upgrade. Live Upgrade works by creating a copy of the running system (called a boot environment), patching the copy, then rebooting into the new system. There is no risk to the running system while patches are being installed – because they are applied to the alternate boot environment – which means the system can continue to run while patches are being installed. And, because the original boot environment is untouched, if any patches do cause problems, we can simply reboot into the old boot environment and diagnose the problem from there. Live Upgrade can even be used to upgrade Solaris itself to a new version – meaning downtime for an operating system upgrade is reduced to a single reboot. If the upgrade fails, simply reboot into the old boot environment.

There are some prequisites to use Live Upgrade. Most importantly, there needs to be free disk space to create the new boot environment on. There are two common ways to do this. #1, Split the root mirror, and #2, allocate unused disk space for the new environment.

#1: Split the root mirror

Most servers have mirrored root disks, so failure of one disk doesn’t cause downtime. If the root disk has been mirrored using Solaris Volume Manager (the software RAID functionality in Solaris), Live Upgrade has built-in support for creating a copy of the existing environment by detaching one side of the mirror. This turns a single device (a mirror of two disks) into two new devices: the individual disks. Because both sides of the mirror are already identical, this method also means there’s no need for a lengthy copy of the current root filesystem.

The downside of this method is that splitting the mirror means it’s no longer redundant. If the disk containing the new boot environment fails during the Live Upgrade process, it can’t be restarted until a replacement disk has been installed. Even worse, if the disk containing the current boot environment fails, the entire system will probably need to be reinstalled. Failure of a disk during a 30-minute window is perhaps unlikely, but the possibility needs to be considered.

On x86 systems, there’s another disadvantage to this method: since the new boot environment is on a different disk, the boot loader (e.g. GRUB) has to be relocated to the new disk, and the BIOS must be told to boot from this new disk. This is easy to forget, which causes surprising problems. (On SPARC systems, Live Upgrade can automatically change the default boot device in the system’s firmware.)

#2: Allocate unused space for Live Upgrade

In this method, the root mirror (if there is one) remains intact. Instead of detaching a device to create space, the system is partitioned at installed time to contain an additional unused slice which is the same size as the root slice. When Live Upgrade is invoked, it copies the current root filesystem into the unused partition. When it’s complete, the unused partition is the new root filesystem, and the existing partition becomes unused.

The most obvious downside of this method is that it needs more space. However, in a typical server configuration, two disks will be dedicated to the root filesystem. Disks these days are large enough (typically at least 72GB) that there’s easily enough space to spare. (We use 20GB root filesystems, which are around 50% used.) The upside, compared to method #1, is that no redundancy is lost during the Live Upgrade operation.

Using Live Upgrade

We use method #2 at the toolserver, for the reasons I mentioned above. So, how do we actually use Live Upgrade to apply patches?

The first step is to create the new boot environment. I’ll demonstrate this on yarrow, one of our database servers. Yarrow’s current root filesystem is /dev/md/dsk/d1. We’ll be creating the new boot environment on /dev/md/dsk/d7. This is pretty simple:

root@yarrow:~#lucreate -n be1 -m /:/dev/md/dsk/d7:ufs -x /aux0

This tells Live Upgrade to create a new boot environment, called be1, whose root (/) filesystem will be /dev/md/dsk/d7. After some time (around an hour), lucreate finishes, and the new environment is ready. Next, we need to apply the patches we want to it.

We use a tool called pca to download the patches we need. Since we want pca to operate on the new boot environment, we mount the environment’s filesystems, and use the -R switch to pca to make it operate on that filesystem.

root@yarrow:~#lumount be1
/.alt.be1
root@yarrow:~#pca -R /.alt.be1 -d
Downloading xref file to /var/tmp/patchdiag.xref
Trying http://sunsolve.sun.com/patchdiag.xref (1/1)
[...]

(We could also use pca -l first, which would list the patches it’s going to download.)

After pca is finished (it produces more output while it’s running, which I haven’t shown here), the patches we need to install on be1 are now in /root/patches, because that’s where we configured pca to download them to. Next, we extract the patches (which come in .zip files):

root@yarrow:~root/patches#ls
119060-41.zip   124864-04.zip   126510-05.zip   127144-03.zip   127965-05.zip   137018-02.zip   138049-01.zip
119961-03.zip   125333-03.zip   127002-04.zip   127923-04.zip   128401-01.zip   137131-01.zip   138076-01.zip
root@yarrow:~root/patches#for p in *.zip; do unzip -q $p; done
root@yarrow:~root/patches#rm *.zip

Now we use luupgrade to apply these patches to the new boot environment:

root@yarrow:~root/patches#luupgrade -t -n be1 -s $PWD *
Validating the contents of the media .
The media contains 14 software patches that can be added.
Mounting the BE .
Adding patches to the BE .
[...]

Lastly, we need to luactivate the new boot environment, so it becomes active at the next reboot:

root@yarrow:~root/patches#luactivate be1
[...]
Activation of boot environment  successful.

Now we reboot, and the new system comes up with our patches installed.

You can read more about Live Upgrade in the Solaris Live Upgrade and Upgrade Planning manual on docs.sun.com (it’s probably a good idea to read that before you try this at home!).