When we first proposed the grant, the plan (which you can read more about at the above link) was to purchase 3 database servers, which we would use to provide a redundant backup for the 3 current servers. However, before we made the purchase, we realised that for the same amount of money, we could purchase 2 database servers, 2 smaller servers and a disk array. The Foundation approved this change, and that’s what we ended up buying.

The purpose of the two small servers and array was to provide redundant service for NFS and LDAP. These services are critical to the platform operation; if either is offline, the entire platform is down. Previously, both were hosted on a single server (hyacinth), which meant the entire Toolserver depended on this server being up. As well as hurting reliability, this made it very difficult to do any maintenance on that server.

Now, however, the NFS and LDAP data is stored on the disk array, which is connected to two servers (turnera and damiana) running Solaris Cluster software. If one server breaks, or we need to do maintenance on it, the services are automatically moved to the other server, with no interruption in service. The array itself has two redundant, independent controllers, making failure quite unlikely.

The previous NFS/LDAP server, which is now idle, has exactly the same specification as a database server. We will be using this as the third redundant database (along with the two we purchased with the grant) to provide redundant access to the MySQL databases. More news on that later.

wolfsbane     up   53+10:55,     1 user,   load 53.93, 55.49, 55.22

It doesn’t seem to have noticed that it’s blocked. It’s still hammering away as fast as it can, and getting nothing but 403 in reply. I’ve even added it to robots.txt, but it doesn’t seem to have noticed that either yet. Fortunately, our web server is quite fast at returning 403, so the load is looking much happier:

wolfsbane     up   53+11:58,     4 users,  load 0.68, 1.15, 3.44

After I blocked it, I tried to find a contact at Microsoft to report the problem too—as the spider clearly isn’t behaving like they expect, I thought they might appreciate a warning. Well, I can now report that Live Search really don’t want to be contacted. The closest thing I could find to a contact form, linked from the “troubleshooting problems with msnbot” page, had a list of categories for me to choose from. None of them was even slightly related to search. Some Googling suggested that “msnbot@microsoft.com” might work, but nope (“Returned mail: user unknown”). There’s a feedback link on the MSN front page, but who knows where that would go, and whether the feedback would ever reach someone who could deal with it? (Certainly not me, as they clearly state that they won’t reply to your feedback.)

I gave up in the end. If someone reading this happens to have a contact at Microsoft who would be interested in this issue, please feel free to let them know. Otherwise, I imagine Live Search users will just have to live without the Toolserver.

PS: I know msnbot (supposedly) supports the Crawl-Delay parameter in robots.txt. But given what I’ve seen today, I don’t particularly want to rely on this, even if it does, some day, reload our robots.txt.

The Toolserver journal is now running on WordPress. WordPress is much nicer to use, looks nicer for users, and has a large community of users. It was also very easy to set up, and allowed us to import the old posts from Roller. The only downside is the lack of an up-to-date LDAP integration plugin. However, I don’t think that should be too hard to write…

PS: In case Planet gets confused by the change and re-displays all our old posts–sorry!

The most obvious is to provide a US replica for cache.stable.toolserver.org, the caching proxy in front of the stable server; this will improve WikiMiniAtlas performance for North American users (the only tool currently using the cache). But that’s only a tiny load, so it would be a waste to use the entire server for just that. Other possibilities include moving our webapps (e.g. JIRA) there, which would free up some resources from the server hosting them in Amsterdam. No doubt, there are several other things we could use it for… please feel free to offer suggestions.

One of the main features of the Toolserver is that it holds a copy of the Wikimedia MySQL databases, and allows users (and tools) to run SQL queries on them. The databases are updated in real time, using a MySQL feature called replication. Replication is a way of applying changes to one database — in this case, Wikimedia’s master database — to a slave server, the Toolserver. It works by recording each change on the master to a file, called a binlog (short for “binary log”). The slave server retrieves this file, and applies each change to its own copy of the database, thus bringing it up to date with the master.

If, for some reason, the slave server isn’t replicating, meaning no changes are being applied to it, the slave’s copy of the database will gradually become more and more out of date compared to the master. This difference, usually expressed as a time (e.g. seconds, hours or days) is called the replication lag, or replag. The most common cause of replag is the slave not being able to apply the changes fast enough. For example, in an hour, it might only be able to apply 40 minutes’ worth of changes. This would cause 20 minutes of replag. As the slave catches up (by applying changes faster than real time), the replag decreases until it reaches 0 again.

Another cause of replag is the slave not replicating at all. This is what happened during the recent maintenance; as the servers were offline, no replication was happening. When the platform came back up, there was about 6 days’ worth of replag. For the s1 (en.wikipedia.org) and s2 (de.wikipedia.org and a few others), there was no problem. Although the lag was high, it has caught up now, and things are working normally. However, for s3, an unexpected problem was encountered: someone deleted the binlogs!

Now, the problem with binlogs is that they take up disk space on the master server, and Wikimedia’s s3 master is currently rather short of disk space. To free some space, an admin deleted binlogs older than a couple of days, not realising that the Toolserver still needed them. Since the record of changes to the master no longer exists, it’s impossible for the slave to replicate. To restart replication, we need to dump (export) a copy of the master database, and import it to the slave, replacing the existing copy of the s3 database. This copy will be new enough that binlogs exist from the time of the dump until now, allowing replication to restart. Unfortunately, this process takes some time to do, and the database is unavailable while it’s happening. Shortly (hopefully, in less than a month), we will be adding a new database server, and when that happens, a dump/import is required anyway. To avoid having to do this twice, we decided to wait until the new server is ready before re-importing s3.

From a users’ point of view, the most obvious effect of this will be that tools which query wikis on the s3 database will return outdated information until the problem is fixed. We realise this is rather inconvenient; sorry.

This has happened a couple time before, but hopefully won’t happen again; after this, I created a way for Wikimedia admins to easily see what binlogs the Toolserver needs, so they can avoid deleting them.

Until now we used something called mod_suphp. This is an Apache module which receives PHP requests, and handles them using suexec and the CGI PHP binary, php-cgi. While this works fine, it creates a large overhead on every request: one fork() and exec() to invoke suexec, and another exec for suexec to run php-cgi.

Fortunately, PHP supports a CGI-replacement called FastCGI. FastCGI runs requests using a persistent daemon process; the web server connects to the process for each request. Since there’s no forking, this is much faster than traditional CGI.

Even though PHP supports FastCGI, neither Apache nor PHP support per-user FastCGI processes — because we want each request to run as the user who owns the script, a seperate PHP daemon is needed for each user. While we could run a seperate PHP for every user, it would use a lot of resources for no reason (we have 300+ users, not all of whom use PHP). The web server configuration to send each request to the right PHP would also be error-prone and difficult to maintain.

Instead, I wrote switchboard. switchboard is a daemon which receives FastCGI requests from the web server, and dynamically creates PHP processes on-demand, running as the appropriate user. After a request is finished, the php-cgi process is kept around to serve the next request for that user. As a side effect, it supports per-user limits on number of processes, which helps to reduce the impact of runaway scripts.

I haven’t done much performance testing, but initial results suggest it significantly reduces load. Right now (due to bugs in earlier versions) it’s not enabled by default for PHP scripts, which makes it harder to see the difference, but once it’s been stable for a while, I’ll probably make it the default handler for PHP.

For more information, and downloads: http://www.flyingparchment.org.uk/pages/switchboard

Unfortunately, when they arrived, we realised we’d forgotten to buy DRAC cards, so we couldn’t use them. (Dell Remote Access Controller is a management card that allows remote power control and serial console access to servers.) Last week, we finally got two DRAC cards installed, so I began setting up the new servers. Mostly, this was pretty simple. A few minor problems:

• The Debian Linux installer doesn’t support the Broadcom NetExtreme II network card, because it requires a so-called “non-free” firmware to be loaded. I fixed this by creating a custom initrd containing the firmware file, after which installation was relatively simple.
• Solaris does support the NetExtreme II, but it doesn’t support the PERC6/i RAID controller. Again, the fix for this was building a custom install miniroot containing the LSI driver for this card (mega_sas).
• Serial console redirection on these servers defaults to “continue after POST”, which has an unfortunate interaction when GRUB is configured to use the serial console; the server appears to hang after POST. This is easy to change, but annoying if you don’t notice it.

Since we now have two user servers, the next issue was sharing user account information between them. The simplest way to do this is just sycning the account data (/etc/passwd, /etc/shadow and /etc/group) between the two servers. There are a few problems with this, though: you have to wait until the next sync for updates to take effect on both servers; and any differences between the two on each server (for example, Debian likes to add new users during package installations) will be lost.

So, the two common options for shared account information are NIS and LDAP. NIS is an older system that provides basic key/value maps (e.g. looking up the username in the ‘passwd’ map, and getting the corresponding entry from /etc/passwd). On the other hand, LDAP provides a database of objects, each of which can have any number of user-defined attributes. The advantage of LDAP is that each user object can have additional information beyond POSIX account information (for example, email accounts). I decided to use LDAP for the toolserver, mainly for this reason.

The LDAP server I used was Sun Directory Server Enterprise Edition. Despite the name, this is a fairly lightweight LDAP server, with good documentation. In the past I’ve used OpenLDAP, but I found it quite fiddly to configure, and not so well documented. The Linux version of DSEE is only supported on RedHat and SUSE systems, but it installed and ran fine on Debian.

Importing the existing account data to LDAP using the PADL LDAP MigrationTools was quite straightforward. For some reason, Debian has two different LDAP NSS modules; libnss-ldap and libnss-ldapd. I chose the latter, which uses a daemon (nslcd) to proxy requests between user applications and the LDAP server. After configuring NSS and PAM to use LDAP, everything seems to be working.

The new stable server was simpler to set up, as it’s replacing the existing server rather than augmenting it. Installing Solaris and the web server was straightforward; all that was left was copying a few configuration files from the old stable server.

One thing I did differently was using pkgbuild for installing software packages, rather than compiling by hand. pkgbuild builds Solaris packages from RPM spec files; as specs contain all the information required to build a package in a single file, it’s easy to create repositories of spec files which can be built using pkgbuild. This is much easier to manage than the old stable server, where things were simply built and installed without using packages. And, as there’s already a large existing repository of such specs (spec-files-extra) to work from, it’s easy to add new software.

These new servers should provide enough capacity for the toolserver to last a long time, as well as improving both interactive performance and web serving. (But look out for a future post about switchboard, a fast replacement for suphp using FastCGI — if I ever get around to finishing it.)

Unfortunately, some of these bots are rather excessive in their memory use. In the past, it was usual to see a few interwiki bots using 500MB+ each. Recently, support for caching to disk (instead of using RAM) was added to pywikipediabot. Hopefully, this will help reduce the memory use of these bots. Next on the list: a Java program called “Linky”, which is apparently some kind of IRC bot. There are 9 copies of Linky running on the toolserver, using 100MB each—that’s 900MB, or nearly ⅛th of the system’s total memory—dedicated to just that.

The problem with running a shared server is that users often don’t realise how many resources they’re using. If you’re running a bot at home, using 100MB RAM is probably fine. But when you have to share available resources with other people, you need to start looking more closely at your resource use, and perhaps spend some time reducing it. (If you think about it for a moment, 100MB just for an IRC bot is a little excessive, isn’t it?)

The main technical restriction for memory use on the toolserver is a homemade daemon called slayerd, which starts killing users’ processses when they use more than the limit (currently set at 1GB). This is mainly intended to stop runaway processes than enforce policy—usually an administrator will start killing processes long before 1GB, if they’re noticed. The only way to really fix the problem is to educate users about the problem. Of course, the problem then is treading the line between gentle reminders, and whining ;-)

Live Upgrade

At the Toolserver, we mitigate patching problems with a process known as Live Upgrade. Live Upgrade works by creating a copy of the running system (called a boot environment), patching the copy, then rebooting into the new system. There is no risk to the running system while patches are being installed – because they are applied to the alternate boot environment – which means the system can continue to run while patches are being installed. And, because the original boot environment is untouched, if any patches do cause problems, we can simply reboot into the old boot environment and diagnose the problem from there. Live Upgrade can even be used to upgrade Solaris itself to a new version – meaning downtime for an operating system upgrade is reduced to a single reboot. If the upgrade fails, simply reboot into the old boot environment.

There are some prequisites to use Live Upgrade. Most importantly, there needs to be free disk space to create the new boot environment on. There are two common ways to do this. #1, Split the root mirror, and #2, allocate unused disk space for the new environment.

#1: Split the root mirror

Most servers have mirrored root disks, so failure of one disk doesn’t cause downtime. If the root disk has been mirrored using Solaris Volume Manager (the software RAID functionality in Solaris), Live Upgrade has built-in support for creating a copy of the existing environment by detaching one side of the mirror. This turns a single device (a mirror of two disks) into two new devices: the individual disks. Because both sides of the mirror are already identical, this method also means there’s no need for a lengthy copy of the current root filesystem.

The downside of this method is that splitting the mirror means it’s no longer redundant. If the disk containing the new boot environment fails during the Live Upgrade process, it can’t be restarted until a replacement disk has been installed. Even worse, if the disk containing the current boot environment fails, the entire system will probably need to be reinstalled. Failure of a disk during a 30-minute window is perhaps unlikely, but the possibility needs to be considered.

On x86 systems, there’s another disadvantage to this method: since the new boot environment is on a different disk, the boot loader (e.g. GRUB) has to be relocated to the new disk, and the BIOS must be told to boot from this new disk. This is easy to forget, which causes surprising problems. (On SPARC systems, Live Upgrade can automatically change the default boot device in the system’s firmware.)

#2: Allocate unused space for Live Upgrade

In this method, the root mirror (if there is one) remains intact. Instead of detaching a device to create space, the system is partitioned at installed time to contain an additional unused slice which is the same size as the root slice. When Live Upgrade is invoked, it copies the current root filesystem into the unused partition. When it’s complete, the unused partition is the new root filesystem, and the existing partition becomes unused.

The most obvious downside of this method is that it needs more space. However, in a typical server configuration, two disks will be dedicated to the root filesystem. Disks these days are large enough (typically at least 72GB) that there’s easily enough space to spare. (We use 20GB root filesystems, which are around 50% used.) The upside, compared to method #1, is that no redundancy is lost during the Live Upgrade operation.

Using Live Upgrade

We use method #2 at the toolserver, for the reasons I mentioned above. So, how do we actually use Live Upgrade to apply patches?

The first step is to create the new boot environment. I’ll demonstrate this on yarrow, one of our database servers. Yarrow’s current root filesystem is /dev/md/dsk/d1. We’ll be creating the new boot environment on /dev/md/dsk/d7. This is pretty simple:

root@yarrow:~#lucreate -n be1 -m /:/dev/md/dsk/d7:ufs -x /aux0

This tells Live Upgrade to create a new boot environment, called be1, whose root (/) filesystem will be /dev/md/dsk/d7. After some time (around an hour), lucreate finishes, and the new environment is ready. Next, we need to apply the patches we want to it.

We use a tool called pca to download the patches we need. Since we want pca to operate on the new boot environment, we mount the environment’s filesystems, and use the -R switch to pca to make it operate on that filesystem.

root@yarrow:~#lumount be1
/.alt.be1
root@yarrow:~#pca -R /.alt.be1 -d
Downloading xref file to /var/tmp/patchdiag.xref
Trying http://sunsolve.sun.com/patchdiag.xref (1/1)
[...]

(We could also use pca -l first, which would list the patches it’s going to download.)

After pca is finished (it produces more output while it’s running, which I haven’t shown here), the patches we need to install on be1 are now in /root/patches, because that’s where we configured pca to download them to. Next, we extract the patches (which come in .zip files):

root@yarrow:~root/patches#ls
119060-41.zip   124864-04.zip   126510-05.zip   127144-03.zip   127965-05.zip   137018-02.zip   138049-01.zip
119961-03.zip   125333-03.zip   127002-04.zip   127923-04.zip   128401-01.zip   137131-01.zip   138076-01.zip
root@yarrow:~root/patches#for p in *.zip; do unzip -q $p; done
root@yarrow:~root/patches#rm *.zip

Now we use luupgrade to apply these patches to the new boot environment:

root@yarrow:~root/patches#luupgrade -t -n be1 -s $PWD *
Validating the contents of the media .
The media contains 14 software patches that can be added.
Mounting the BE .
Adding patches to the BE .
[...]

Lastly, we need to luactivate the new boot environment, so it becomes active at the next reboot:

root@yarrow:~root/patches#luactivate be1
[...]
Activation of boot environment  successful.

Now we reboot, and the new system comes up with our patches installed.

You can read more about Live Upgrade in the Solaris Live Upgrade and Upgrade Planning manual on docs.sun.com (it’s probably a good idea to read that before you try this at home!).