When we first proposed the grant, the plan (which you can read more about at the above link) was to purchase 3 database servers, which we would use to provide a redundant backup for the 3 current servers. However, before we made the purchase, we realised that for the same amount of money, we could purchase 2 database servers, 2 smaller servers and a disk array. The Foundation approved this change, and that’s what we ended up buying.

The purpose of the two small servers and array was to provide redundant service for NFS and LDAP. These services are critical to the platform operation; if either is offline, the entire platform is down. Previously, both were hosted on a single server (hyacinth), which meant the entire Toolserver depended on this server being up. As well as hurting reliability, this made it very difficult to do any maintenance on that server.

Now, however, the NFS and LDAP data is stored on the disk array, which is connected to two servers (turnera and damiana) running Solaris Cluster software. If one server breaks, or we need to do maintenance on it, the services are automatically moved to the other server, with no interruption in service. The array itself has two redundant, independent controllers, making failure quite unlikely.

The previous NFS/LDAP server, which is now idle, has exactly the same specification as a database server. We will be using this as the third redundant database (along with the two we purchased with the grant) to provide redundant access to the MySQL databases. More news on that later.

To improve this situation, we plan to buy three new database servers, so we can keep two copies of each database, instead of just one. This way, one copy will remain available when the other breaks, and we will be able to fix things without too much interruption. The new servers will very likely be the same as our other newer database servers, namely, Sun Fire X4250s with 32GB RAM and 16 internal disks with 146 GB each. We hope to have these online some time in September or October. This should greatly improve the availability of live replication, and thus of any tools relying on real time information.

After I was trough the tftp-network-installation-hell again (took me only 2h and a dozen reboots to let the installation starting…), we discovered that the host doesn’t have a hardware-raid. A fast investigation result in that the osm-squid ortelius (which is nearly identical to cassini) and cassini were swap at the hardware installation. Because cassini should run a little copy of the osm-database itself, using it without hardware-raid is no option.

We are unsure what is the best way to handle the situation and how long it will take until cassini can used at the moment.

The toolserver-cluster got two new boxes: daphe, that is going to replace zedler as databasehost of the s2-cluster, and hyacinth that is going to work as host for our /home-directory.

Also they installed new servers for the openstreetmap/wikimedia-cooperation: A openstreetmap-toolserver on that people can play with osm-data (like our toolserver for wikimedia) named cassini, and a squid-proxy named ortelius and a database-host named ptolemy – both to handle the load the wikimedia-projects will create when they will use openstreetmap-maps.

At last two terminal-servers were installed to make it easier for the roots to handle broken servers.

Thanks to Mark and Multichil!

wolfsbane     up   53+10:55,     1 user,   load 53.93, 55.49, 55.22

It doesn’t seem to have noticed that it’s blocked. It’s still hammering away as fast as it can, and getting nothing but 403 in reply. I’ve even added it to robots.txt, but it doesn’t seem to have noticed that either yet. Fortunately, our web server is quite fast at returning 403, so the load is looking much happier:

wolfsbane     up   53+11:58,     4 users,  load 0.68, 1.15, 3.44

After I blocked it, I tried to find a contact at Microsoft to report the problem too—as the spider clearly isn’t behaving like they expect, I thought they might appreciate a warning. Well, I can now report that Live Search really don’t want to be contacted. The closest thing I could find to a contact form, linked from the “troubleshooting problems with msnbot” page, had a list of categories for me to choose from. None of them was even slightly related to search. Some Googling suggested that “msnbot@microsoft.com” might work, but nope (”Returned mail: user unknown”). There’s a feedback link on the MSN front page, but who knows where that would go, and whether the feedback would ever reach someone who could deal with it? (Certainly not me, as they clearly state that they won’t reply to your feedback.)

I gave up in the end. If someone reading this happens to have a contact at Microsoft who would be interested in this issue, please feel free to let them know. Otherwise, I imagine Live Search users will just have to live without the Toolserver.

PS: I know msnbot (supposedly) supports the Crawl-Delay parameter in robots.txt. But given what I’ve seen today, I don’t particularly want to rely on this, even if it does, some day, reload our robots.txt.

The first server will replace Zedler as our database server for the s2 cluster. Zedler is our oldest database server, and has lately been overloaded nearly constantly. We may keep Zedler around with a backup copy of s2, but it will mostlky be idle. We’ll see what use we can put it to later on.

The second server will take Hemlock’s duty of serving the home directories, and it will become the host system of the stable server. This means the stable server becomes virtualized (probably as a Solaris zone). Willow (the current server for stable projects) will then be free, we will probably make it into a second login server where users can run bots. This should take some load off Nightshade.

The thirs server is the “OpenStreetMap Toolserver”: it will be for the OpenStreeMap project what the Toolserver cluster has so far been for Wikimedia projects: a place to play with data and host bots and web applications. The Toolserver rules have been changed to accomodate this.

All in all, we will then have 12 servers in the Toolserver cluster.

The two remaining servers that have been ordered yesterday will also be used for the OpenStreetMap project, but not in the context of the Toolserver. They will be used to integrate interactive maps from OpenStreetMap directly into wikipedia Articles. More information about the OSM integration project is avialable on meta.

The Toolserver journal is now running on WordPress. WordPress is much nicer to use, looks nicer for users, and has a large community of users. It was also very easy to set up, and allowed us to import the old posts from Roller. The only downside is the lack of an up-to-date LDAP integration plugin. However, I don’t think that should be too hard to write…

PS: In case Planet gets confused by the change and re-displays all our old posts–sorry!

The most obvious is to provide a US replica for cache.stable.toolserver.org, the caching proxy in front of the stable server; this will improve WikiMiniAtlas performance for North American users (the only tool currently using the cache). But that’s only a tiny load, so it would be a waste to use the entire server for just that. Other possibilities include moving our webapps (e.g. JIRA) there, which would free up some resources from the server hosting them in Amsterdam. No doubt, there are several other things we could use it for… please feel free to offer suggestions.

One of the main features of the Toolserver is that it holds a copy of the Wikimedia MySQL databases, and allows users (and tools) to run SQL queries on them. The databases are updated in real time, using a MySQL feature called replication. Replication is a way of applying changes to one database — in this case, Wikimedia’s master database — to a slave server, the Toolserver. It works by recording each change on the master to a file, called a binlog (short for “binary log”). The slave server retrieves this file, and applies each change to its own copy of the database, thus bringing it up to date with the master.

If, for some reason, the slave server isn’t replicating, meaning no changes are being applied to it, the slave’s copy of the database will gradually become more and more out of date compared to the master. This difference, usually expressed as a time (e.g. seconds, hours or days) is called the replication lag, or replag. The most common cause of replag is the slave not being able to apply the changes fast enough. For example, in an hour, it might only be able to apply 40 minutes’ worth of changes. This would cause 20 minutes of replag. As the slave catches up (by applying changes faster than real time), the replag decreases until it reaches 0 again.

Another cause of replag is the slave not replicating at all. This is what happened during the recent maintenance; as the servers were offline, no replication was happening. When the platform came back up, there was about 6 days’ worth of replag. For the s1 (en.wikipedia.org) and s2 (de.wikipedia.org and a few others), there was no problem. Although the lag was high, it has caught up now, and things are working normally. However, for s3, an unexpected problem was encountered: someone deleted the binlogs!

Now, the problem with binlogs is that they take up disk space on the master server, and Wikimedia’s s3 master is currently rather short of disk space. To free some space, an admin deleted binlogs older than a couple of days, not realising that the Toolserver still needed them. Since the record of changes to the master no longer exists, it’s impossible for the slave to replicate. To restart replication, we need to dump (export) a copy of the master database, and import it to the slave, replacing the existing copy of the s3 database. This copy will be new enough that binlogs exist from the time of the dump until now, allowing replication to restart. Unfortunately, this process takes some time to do, and the database is unavailable while it’s happening. Shortly (hopefully, in less than a month), we will be adding a new database server, and when that happens, a dump/import is required anyway. To avoid having to do this twice, we decided to wait until the new server is ready before re-importing s3.

From a users’ point of view, the most obvious effect of this will be that tools which query wikis on the s3 database will return outdated information until the problem is fixed. We realise this is rather inconvenient; sorry.

This has happened a couple time before, but hopefully won’t happen again; after this, I created a way for Wikimedia admins to easily see what binlogs the Toolserver needs, so they can avoid deleting them.