2010-06-07: Updating to ejabberd

Interestingly, updating to ejabberd went without any problems (so far), including upgrading the server hardware. Even migration of account data was no problem. The only glitches I saw were that stats were off immediately after import of jabberd14 data and that ejabberdctl crashes with a non-helpful message if epmd TCP port 4369 is blocked by the firewall.

Memory consumption of ejabberd is higher than jabberd14 but 280/330MB res/virt RAM for about 150/80 c2s/s2s connections without any configuration tweaks is okay for me. CPU usage is negligible.

2010-05-04: Short Survey of XMPP Server Implementations

The list of XMPP servers on xmpp.org basically lists every server software available. Unfortunately, this also includes unmaintained and obscure servers, without any additional information. Then again, remarks on this topic are probably difficult on an official site ;-).

My 10 ct concerning some server implementations:

A query on the servers jabber.fsinf.de is typically connected to resulted in...

So ejabberd seems the way to go for now.

2010-05-03: Jabber Server Update Woes

On jabber.fsinf.de (about 150 users concurrently online max, about 500 active accounts, 14k average roster size including avatar data etc. for active accounts), jabberd 1.4.4 has been running for a long time. Apart from its lack of features (no s2s encryption, no STARTTLS, etc.), it ran reasonably stable and consumed few resources (it's only a 256MB machine). From time to time a restart was needed as it would fail to set up s2s connections eventually, but this hardly mattered.

However, sometime in April 2010, s2s connections to jabber.org failed consistently. This was probably caused by non-RFC s2s messages generated by jabberd 1.4.4 that the M-Link software on jabber.org's side decided to drop recently. Apparently, it was time to upgrade.

Since these days I don't have much time and motivation to do server administration, I simply chose to use the jabberd14 1.6.1 Debian package. On first tests, the new jabberd failed to open any incoming TCP ports, without any helpful error message. After digging around some time, it turned out that the Debian package was compiled with IPv6 enabled. However, in contrast to for example Apache or probably almost every other Internet daemon, in case of jabberd14 this does not mean IPv6 kernel support is optional - IPv6 methods are used for setting up IPv4 connections then, too. The kernel used on jabber.fsinf.de is a custom grsecurity kernel with no unneeded features compiled in though, especially not IPv6. Well, recompile with IPv6 enabled - and file a Debian bug report.

After the update, I got strange "subscription removed" messages by some of my contacts. As it turned out much later, this was caused by a chain of bugs starting long in the past: Some contacts have used the Konverse Jabber client (probably years ago). This wrote some configuration data to the account. However, it used some XML namespaces that were not declared properly. jabberd 1.4.4, not supporting namespaces, ignored that. jabberd14 1.6 does support namespaces and apparently failed to read the account data. Instead of marking these accounts as suspended, it marked them as nonexistent though, and sent unsubscribes whenever incoming messages referred to them. Result: People having used some obscure Jabber client a long time ago were unsubscribed from all their contacts and not able to log in until the account XML was fixed manually. At least only very few people were affected by this.

After a bit more testing, it turned out that legacy SSL for some reason stops accepting new connections randomly. Workaround: Disable legacy SSL in jabberd14 1.6 and use stunnel instead.

But we have two more problems to go:

  1. Some s2s connections fail, probably due to strange peer certificates not accepted by GnuTLS.
  2. The server crashes randomly, perhaps twice a day, with segmentation faults.

What do we learn from this?