So, a while ago, Sun donated some servers to Wikimedia. We managed to earmark one of them for the Toolserver, but for various reasons, it couldn’t be delivered to Amsterdam, so it ended up at our Tampa facility. It’s been sitting there ever since, until yesterday, when we managed to rack it and set up its OS and array. The server is a Sun Fire X4150, with 2 quad-core 2.8GHz CPUs, 32GB RAM, 4 146GB system disks, and an external array with 12 15′000 rpm 146GB SAS disks. Originally, it was meant to be a database server, but there’s not much point putting a single database server at Tampa, so now we’re looking for other uses.

The most obvious is to provide a US replica for cache.stable.toolserver.org, the caching proxy in front of the stable server; this will improve WikiMiniAtlas performance for North American users (the only tool currently using the cache). But that’s only a tiny load, so it would be a waste to use the entire server for just that. Other possibilities include moving our webapps (e.g. JIRA) there, which would free up some resources from the server hosting them in Amsterdam. No doubt, there are several other things we could use it for… please feel free to offer suggestions.

Many people have noticed that since the maintenance a couple of weeks ago, replication of s3 (the cluster that holds frwiki, and most of the smaller wikis) is halted, which means Toolserver tools are returning outdated information. So, what is replication, and why isn’t it working?

One of the main features of the Toolserver is that it holds a copy of the Wikimedia MySQL databases, and allows users (and tools) to run SQL queries on them. The databases are updated in real time, using a MySQL feature called replication. Replication is a way of applying changes to one database — in this case, Wikimedia’s master database — to a slave server, the Toolserver. It works by recording each change on the master to a file, called a binlog (short for “binary log”). The slave server retrieves this file, and applies each change to its own copy of the database, thus bringing it up to date with the master.

If, for some reason, the slave server isn’t replicating, meaning no changes are being applied to it, the slave’s copy of the database will gradually become more and more out of date compared to the master. This difference, usually expressed as a time (e.g. seconds, hours or days) is called the replication lag, or replag. The most common cause of replag is the slave not being able to apply the changes fast enough. For example, in an hour, it might only be able to apply 40 minutes’ worth of changes. This would cause 20 minutes of replag. As the slave catches up (by applying changes faster than real time), the replag decreases until it reaches 0 again.

Another cause of replag is the slave not replicating at all. This is what happened during the recent maintenance; as the servers were offline, no replication was happening. When the platform came back up, there was about 6 days’ worth of replag. For the s1 (en.wikipedia.org) and s2 (de.wikipedia.org and a few others), there was no problem. Although the lag was high, it has caught up now, and things are working normally. However, for s3, an unexpected problem was encountered: someone deleted the binlogs!

Now, the problem with binlogs is that they take up disk space on the master server, and Wikimedia’s s3 master is currently rather short of disk space. To free some space, an admin deleted binlogs older than a couple of days, not realising that the Toolserver still needed them. Since the record of changes to the master no longer exists, it’s impossible for the slave to replicate. To restart replication, we need to dump (export) a copy of the master database, and import it to the slave, replacing the existing copy of the s3 database. This copy will be new enough that binlogs exist from the time of the dump until now, allowing replication to restart. Unfortunately, this process takes some time to do, and the database is unavailable while it’s happening. Shortly (hopefully, in less than a month), we will be adding a new database server, and when that happens, a dump/import is required anyway. To avoid having to do this twice, we decided to wait until the new server is ready before re-importing s3.

From a users’ point of view, the most obvious effect of this will be that tools which query wikis on the s3 database will return outdated information until the problem is fixed. We realise this is rather inconvenient; sorry.

This has happened a couple time before, but hopefully won’t happen again; after this, I created a way for Wikimedia admins to easily see what binlogs the Toolserver needs, so they can avoid deleting them.