After I was trough the tftp-network-installation-hell again (took me only 2h and a dozen reboots to let the installation starting…), we discovered that the host doesn’t have a hardware-raid. A fast investigation result in that the osm-squid ortelius (which is nearly identical to cassini) and cassini were swap at the hardware installation. Because cassini should run a little copy of the osm-database itself, using it without hardware-raid is no option.

We are unsure what is the best way to handle the situation and how long it will take until cassini can used at the moment.

The toolserver-cluster got two new boxes: daphe, that is going to replace zedler as databasehost of the s2-cluster, and hyacinth that is going to work as host for our /home-directory.

Also they installed new servers for the openstreetmap/wikimedia-cooperation: A openstreetmap-toolserver on that people can play with osm-data (like our toolserver for wikimedia) named cassini, and a squid-proxy named ortelius and a database-host named ptolemy – both to handle the load the wikimedia-projects will create when they will use openstreetmap-maps.

At last two terminal-servers were installed to make it easier for the roots to handle broken servers.

Thanks to Mark and Multichil!

The first server will replace Zedler as our database server for the s2 cluster. Zedler is our oldest database server, and has lately been overloaded nearly constantly. We may keep Zedler around with a backup copy of s2, but it will mostlky be idle. We’ll see what use we can put it to later on.

The second server will take Hemlock’s duty of serving the home directories, and it will become the host system of the stable server. This means the stable server becomes virtualized (probably as a Solaris zone). Willow (the current server for stable projects) will then be free, we will probably make it into a second login server where users can run bots. This should take some load off Nightshade.

The thirs server is the “OpenStreetMap Toolserver”: it will be for the OpenStreeMap project what the Toolserver cluster has so far been for Wikimedia projects: a place to play with data and host bots and web applications. The Toolserver rules have been changed to accomodate this.

All in all, we will then have 12 servers in the Toolserver cluster.

The two remaining servers that have been ordered yesterday will also be used for the OpenStreetMap project, but not in the context of the Toolserver. They will be used to integrate interactive maps from OpenStreetMap directly into wikipedia Articles. More information about the OSM integration project is avialable on meta.

The most obvious is to provide a US replica for cache.stable.toolserver.org, the caching proxy in front of the stable server; this will improve WikiMiniAtlas performance for North American users (the only tool currently using the cache). But that’s only a tiny load, so it would be a waste to use the entire server for just that. Other possibilities include moving our webapps (e.g. JIRA) there, which would free up some resources from the server hosting them in Amsterdam. No doubt, there are several other things we could use it for… please feel free to offer suggestions.

One of the main features of the Toolserver is that it holds a copy of the Wikimedia MySQL databases, and allows users (and tools) to run SQL queries on them. The databases are updated in real time, using a MySQL feature called replication. Replication is a way of applying changes to one database — in this case, Wikimedia’s master database — to a slave server, the Toolserver. It works by recording each change on the master to a file, called a binlog (short for “binary log”). The slave server retrieves this file, and applies each change to its own copy of the database, thus bringing it up to date with the master.

If, for some reason, the slave server isn’t replicating, meaning no changes are being applied to it, the slave’s copy of the database will gradually become more and more out of date compared to the master. This difference, usually expressed as a time (e.g. seconds, hours or days) is called the replication lag, or replag. The most common cause of replag is the slave not being able to apply the changes fast enough. For example, in an hour, it might only be able to apply 40 minutes’ worth of changes. This would cause 20 minutes of replag. As the slave catches up (by applying changes faster than real time), the replag decreases until it reaches 0 again.

Another cause of replag is the slave not replicating at all. This is what happened during the recent maintenance; as the servers were offline, no replication was happening. When the platform came back up, there was about 6 days’ worth of replag. For the s1 (en.wikipedia.org) and s2 (de.wikipedia.org and a few others), there was no problem. Although the lag was high, it has caught up now, and things are working normally. However, for s3, an unexpected problem was encountered: someone deleted the binlogs!

Now, the problem with binlogs is that they take up disk space on the master server, and Wikimedia’s s3 master is currently rather short of disk space. To free some space, an admin deleted binlogs older than a couple of days, not realising that the Toolserver still needed them. Since the record of changes to the master no longer exists, it’s impossible for the slave to replicate. To restart replication, we need to dump (export) a copy of the master database, and import it to the slave, replacing the existing copy of the s3 database. This copy will be new enough that binlogs exist from the time of the dump until now, allowing replication to restart. Unfortunately, this process takes some time to do, and the database is unavailable while it’s happening. Shortly (hopefully, in less than a month), we will be adding a new database server, and when that happens, a dump/import is required anyway. To avoid having to do this twice, we decided to wait until the new server is ready before re-importing s3.

From a users’ point of view, the most obvious effect of this will be that tools which query wikis on the s3 database will return outdated information until the problem is fixed. We realise this is rather inconvenient; sorry.

This has happened a couple time before, but hopefully won’t happen again; after this, I created a way for Wikimedia admins to easily see what binlogs the Toolserver needs, so they can avoid deleting them.

Until now we used something called mod_suphp. This is an Apache module which receives PHP requests, and handles them using suexec and the CGI PHP binary, php-cgi. While this works fine, it creates a large overhead on every request: one fork() and exec() to invoke suexec, and another exec for suexec to run php-cgi.

Fortunately, PHP supports a CGI-replacement called FastCGI. FastCGI runs requests using a persistent daemon process; the web server connects to the process for each request. Since there’s no forking, this is much faster than traditional CGI.

Even though PHP supports FastCGI, neither Apache nor PHP support per-user FastCGI processes — because we want each request to run as the user who owns the script, a seperate PHP daemon is needed for each user. While we could run a seperate PHP for every user, it would use a lot of resources for no reason (we have 300+ users, not all of whom use PHP). The web server configuration to send each request to the right PHP would also be error-prone and difficult to maintain.

Instead, I wrote switchboard. switchboard is a daemon which receives FastCGI requests from the web server, and dynamically creates PHP processes on-demand, running as the appropriate user. After a request is finished, the php-cgi process is kept around to serve the next request for that user. As a side effect, it supports per-user limits on number of processes, which helps to reduce the impact of runaway scripts.

I haven’t done much performance testing, but initial results suggest it significantly reduces load. Right now (due to bugs in earlier versions) it’s not enabled by default for PHP scripts, which makes it harder to see the difference, but once it’s been stable for a while, I’ll probably make it the default handler for PHP.

For more information, and downloads: http://www.flyingparchment.org.uk/pages/switchboard

Unfortunately, when they arrived, we realised we’d forgotten to buy DRAC cards, so we couldn’t use them. (Dell Remote Access Controller is a management card that allows remote power control and serial console access to servers.) Last week, we finally got two DRAC cards installed, so I began setting up the new servers. Mostly, this was pretty simple. A few minor problems:

• The Debian Linux installer doesn’t support the Broadcom NetExtreme II network card, because it requires a so-called “non-free” firmware to be loaded. I fixed this by creating a custom initrd containing the firmware file, after which installation was relatively simple.
• Solaris does support the NetExtreme II, but it doesn’t support the PERC6/i RAID controller. Again, the fix for this was building a custom install miniroot containing the LSI driver for this card (mega_sas).
• Serial console redirection on these servers defaults to “continue after POST”, which has an unfortunate interaction when GRUB is configured to use the serial console; the server appears to hang after POST. This is easy to change, but annoying if you don’t notice it.

Since we now have two user servers, the next issue was sharing user account information between them. The simplest way to do this is just sycning the account data (/etc/passwd, /etc/shadow and /etc/group) between the two servers. There are a few problems with this, though: you have to wait until the next sync for updates to take effect on both servers; and any differences between the two on each server (for example, Debian likes to add new users during package installations) will be lost.

So, the two common options for shared account information are NIS and LDAP. NIS is an older system that provides basic key/value maps (e.g. looking up the username in the ‘passwd’ map, and getting the corresponding entry from /etc/passwd). On the other hand, LDAP provides a database of objects, each of which can have any number of user-defined attributes. The advantage of LDAP is that each user object can have additional information beyond POSIX account information (for example, email accounts). I decided to use LDAP for the toolserver, mainly for this reason.

The LDAP server I used was Sun Directory Server Enterprise Edition. Despite the name, this is a fairly lightweight LDAP server, with good documentation. In the past I’ve used OpenLDAP, but I found it quite fiddly to configure, and not so well documented. The Linux version of DSEE is only supported on RedHat and SUSE systems, but it installed and ran fine on Debian.

Importing the existing account data to LDAP using the PADL LDAP MigrationTools was quite straightforward. For some reason, Debian has two different LDAP NSS modules; libnss-ldap and libnss-ldapd. I chose the latter, which uses a daemon (nslcd) to proxy requests between user applications and the LDAP server. After configuring NSS and PAM to use LDAP, everything seems to be working.

The new stable server was simpler to set up, as it’s replacing the existing server rather than augmenting it. Installing Solaris and the web server was straightforward; all that was left was copying a few configuration files from the old stable server.

One thing I did differently was using pkgbuild for installing software packages, rather than compiling by hand. pkgbuild builds Solaris packages from RPM spec files; as specs contain all the information required to build a package in a single file, it’s easy to create repositories of spec files which can be built using pkgbuild. This is much easier to manage than the old stable server, where things were simply built and installed without using packages. And, as there’s already a large existing repository of such specs (spec-files-extra) to work from, it’s easy to add new software.

These new servers should provide enough capacity for the toolserver to last a long time, as well as improving both interactive performance and web serving. (But look out for a future post about switchboard, a fast replacement for suphp using FastCGI — if I ever get around to finishing it.)

Unfortunately, some of these bots are rather excessive in their memory use. In the past, it was usual to see a few interwiki bots using 500MB+ each. Recently, support for caching to disk (instead of using RAM) was added to pywikipediabot. Hopefully, this will help reduce the memory use of these bots. Next on the list: a Java program called “Linky”, which is apparently some kind of IRC bot. There are 9 copies of Linky running on the toolserver, using 100MB each—that’s 900MB, or nearly ⅛th of the system’s total memory—dedicated to just that.

The problem with running a shared server is that users often don’t realise how many resources they’re using. If you’re running a bot at home, using 100MB RAM is probably fine. But when you have to share available resources with other people, you need to start looking more closely at your resource use, and perhaps spend some time reducing it. (If you think about it for a moment, 100MB just for an IRC bot is a little excessive, isn’t it?)

The main technical restriction for memory use on the toolserver is a homemade daemon called slayerd, which starts killing users’ processses when they use more than the limit (currently set at 1GB). This is mainly intended to stop runaway processes than enforce policy—usually an administrator will start killing processes long before 1GB, if they’re noticed. The only way to really fix the problem is to educate users about the problem. Of course, the problem then is treading the line between gentle reminders, and whining ;-)

Live Upgrade

At the Toolserver, we mitigate patching problems with a process known as Live Upgrade. Live Upgrade works by creating a copy of the running system (called a boot environment), patching the copy, then rebooting into the new system. There is no risk to the running system while patches are being installed – because they are applied to the alternate boot environment – which means the system can continue to run while patches are being installed. And, because the original boot environment is untouched, if any patches do cause problems, we can simply reboot into the old boot environment and diagnose the problem from there. Live Upgrade can even be used to upgrade Solaris itself to a new version – meaning downtime for an operating system upgrade is reduced to a single reboot. If the upgrade fails, simply reboot into the old boot environment.

There are some prequisites to use Live Upgrade. Most importantly, there needs to be free disk space to create the new boot environment on. There are two common ways to do this. #1, Split the root mirror, and #2, allocate unused disk space for the new environment.

#1: Split the root mirror

Most servers have mirrored root disks, so failure of one disk doesn’t cause downtime. If the root disk has been mirrored using Solaris Volume Manager (the software RAID functionality in Solaris), Live Upgrade has built-in support for creating a copy of the existing environment by detaching one side of the mirror. This turns a single device (a mirror of two disks) into two new devices: the individual disks. Because both sides of the mirror are already identical, this method also means there’s no need for a lengthy copy of the current root filesystem.

The downside of this method is that splitting the mirror means it’s no longer redundant. If the disk containing the new boot environment fails during the Live Upgrade process, it can’t be restarted until a replacement disk has been installed. Even worse, if the disk containing the current boot environment fails, the entire system will probably need to be reinstalled. Failure of a disk during a 30-minute window is perhaps unlikely, but the possibility needs to be considered.

On x86 systems, there’s another disadvantage to this method: since the new boot environment is on a different disk, the boot loader (e.g. GRUB) has to be relocated to the new disk, and the BIOS must be told to boot from this new disk. This is easy to forget, which causes surprising problems. (On SPARC systems, Live Upgrade can automatically change the default boot device in the system’s firmware.)

#2: Allocate unused space for Live Upgrade

In this method, the root mirror (if there is one) remains intact. Instead of detaching a device to create space, the system is partitioned at installed time to contain an additional unused slice which is the same size as the root slice. When Live Upgrade is invoked, it copies the current root filesystem into the unused partition. When it’s complete, the unused partition is the new root filesystem, and the existing partition becomes unused.

The most obvious downside of this method is that it needs more space. However, in a typical server configuration, two disks will be dedicated to the root filesystem. Disks these days are large enough (typically at least 72GB) that there’s easily enough space to spare. (We use 20GB root filesystems, which are around 50% used.) The upside, compared to method #1, is that no redundancy is lost during the Live Upgrade operation.

Using Live Upgrade

We use method #2 at the toolserver, for the reasons I mentioned above. So, how do we actually use Live Upgrade to apply patches?

The first step is to create the new boot environment. I’ll demonstrate this on yarrow, one of our database servers. Yarrow’s current root filesystem is /dev/md/dsk/d1. We’ll be creating the new boot environment on /dev/md/dsk/d7. This is pretty simple:

root@yarrow:~#lucreate -n be1 -m /:/dev/md/dsk/d7:ufs -x /aux0

This tells Live Upgrade to create a new boot environment, called be1, whose root (/) filesystem will be /dev/md/dsk/d7. After some time (around an hour), lucreate finishes, and the new environment is ready. Next, we need to apply the patches we want to it.

We use a tool called pca to download the patches we need. Since we want pca to operate on the new boot environment, we mount the environment’s filesystems, and use the -R switch to pca to make it operate on that filesystem.

root@yarrow:~#lumount be1
/.alt.be1
root@yarrow:~#pca -R /.alt.be1 -d
Downloading xref file to /var/tmp/patchdiag.xref
Trying http://sunsolve.sun.com/patchdiag.xref (1/1)
[...]

(We could also use pca -l first, which would list the patches it’s going to download.)

After pca is finished (it produces more output while it’s running, which I haven’t shown here), the patches we need to install on be1 are now in /root/patches, because that’s where we configured pca to download them to. Next, we extract the patches (which come in .zip files):

root@yarrow:~root/patches#ls
119060-41.zip   124864-04.zip   126510-05.zip   127144-03.zip   127965-05.zip   137018-02.zip   138049-01.zip
119961-03.zip   125333-03.zip   127002-04.zip   127923-04.zip   128401-01.zip   137131-01.zip   138076-01.zip
root@yarrow:~root/patches#for p in *.zip; do unzip -q $p; done
root@yarrow:~root/patches#rm *.zip

Now we use luupgrade to apply these patches to the new boot environment:

root@yarrow:~root/patches#luupgrade -t -n be1 -s $PWD *
Validating the contents of the media .
The media contains 14 software patches that can be added.
Mounting the BE .
Adding patches to the BE .
[...]

Lastly, we need to luactivate the new boot environment, so it becomes active at the next reboot:

root@yarrow:~root/patches#luactivate be1
[...]
Activation of boot environment  successful.

Now we reboot, and the new system comes up with our patches installed.

You can read more about Live Upgrade in the Solaris Live Upgrade and Upgrade Planning manual on docs.sun.com (it’s probably a good idea to read that before you try this at home!).

Unfortunately, although Roller itself was easy to set up, making it work properly with Crowd took a bit more effort. Firstly, there are three ways to integrate Roller with Crowd: LDAP (if your Crowd directory is using LDAP), the Crowd connector for JAAS (a standardised security framework for Java), and the Crowd connector for Acegi Security, the security framework Roller itself uses. After a lot of trial and error, the easiest to set up of all of these appears to be Acegi. So, here is how I did it:

Firstly, make sure you’re using Crowd 1.4.2 or later. Earlier versions (at least 1.4, which I tried) have a bug in the Acegi connector that makes it unusable. You don’t need to upgrade Crowd itself, just make sure you have the 1.4.2 client libraries. Copy the Crowd connector to Roller’s WEB-INF/lib:

$ cp $CROWD_DIR/client/lib/* WEB-INF/lib/
$ cp $CROWD_DIR/client/crowd-integration-client-1.4.2.jar WEB-INF/lib/

The Crowd client libraries include newer versions of some libraries which Roller also includes. If you don’t remove these older versions, Crowd will try to use them, and things won’t work:

$ rm WEB-INF/lib/commons-httpclient-2.0.2.jar
$ rm WEB-INF/lib/ehcache-1.1.jar

Now you can follow the Crowd Acegi set up instructions. When it talks about your Acegi Security configuration, this is Roller’s WEB-INF/security.xml file.

After you’re done, a little bit more work is still needed. Firstly, the applicationContext-CrowdClient.xml from Crowd didn’t work for me; it needs a little bit of editing. Extract the file from the Crowd jar:

$ jar xvf lib/crowd-integration-client-1.4.2.jar applicationContext-CrowdClient.xml

The top of the file should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

Change that to read:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN"
"http://www.springframework.org/dtd/spring-beans.dtd">
<beans>

Then edit web.xml so it refers to /WEB-INF/applicationContext-CrowdClient.xml instead of the one in classpath:/. My web.xml looks like this:

<context-param>
<param-name>contextConfigLocation</param-name>
<param-value>
/WEB-INF/applicationContext-CrowdClient.xml
/WEB-INF/security.xml
</param-value>
</context-param>

Finally, you need to map Crowd groups to Roller groups. Create two new groups (e.g. roller-editor and roller-admin) in Crowd. Then edit security.xml again, and find the “AUTHENTICATION” section. It should have several lines that look like this:

/roller-ui/login-redirect**=admin,editor

On each line, change “admin” to “ROLE_roller-admin“, and “editor” to “ROLE_roller-editor“.

That’s it! Hopefully, your Roller installation should now authenticate from Crowd. Easy, huh? (Remember to re-apply these customisations each time you upgrade Roller…)