The server migration seems to be close to complete at this point.. FishEye is back online, and SpamAssassin is once again fully operational. I’ve also had time to track down and fix most of the known issues.. for instance, Tomcat would die for no reason with an OutOfMemory error (turns out it needed a larger PermSize, probably because it’s using a 64-bit JDK). I’ve also fixed a network card driver issue that caused the system to slow down quite a bit, so I am hoping that the system is stabilizing. FTP users have been given new passwords, so if you can’t login just send me an IM or email.
There are still a few issues to be fixed, so should it happen that a service is down for a minute, please bear with me. If anything is down for more than a few minutes, feel free to start complaining again ;-)
JIRA and Confluence are back online, and I’ve restored the missing plugins and corrected a few minor issues. FishEye should be back tomorrow too.
Subversion is also back, but the Gentle repository has a new URL (it is now http://www.mertner.com/svn/gentle instead of http://www.mertner.com/svn/repos). Note that the “projects” folder has been stripped on import. This means that you will have to relocate your working copy to the new URL. If you previously checked out /svn/repos/projects/gentle then you should relocate this to /svn/gentle/gentle (I tried to strip out the extra gentle too, but then Subversion would not be able to import it for some reason). The NProf repository has not moved and is still available at /svn/nprof.
Apparently the PHP developers have decided that using recode along with mysql is a bad thing, which has broken international characters on some domains. I’ve tried convincing PHP of compiling in support for both anyway, since it used to work just fine, but alas, it just wont no matter what.
Mail should now be operational again across all domains. I ran into a ton of problems with the Exim configuration file on the new system (which uses LDAP and virtual mail accounts rather than system users), and it has taken most of today to iron out all of the problems this caused.
There are still a few quirks: SpamAssassin is somehow incapable of setting its home dir, so it cannot find the bayes database (this is not critical, but does mean that more spam than usual will be slipping through, at least until it gets fixed). Some users have received new login and/or password. If you cannot login, please catch me on IM to get your updated account information.
Apart from webmail still not being online, email should now be in perfect working order. Let me know if you discover anything that might indicate a problem.
Now, on to the next problem…
Of course, nothing is ever as easy as expected. I won’t go into too much detail at this hour, but as you can easily observe the server upgrade is still far from complete. I’ll post status updates at regular intervals tomorrow (that is, later today, but after catching some sleep) as the various services come online (web and email support is almost, but not quite, working – sigh).
At this point it seems unlikely that I’ll be able to get all of the Java-based services (Confluence, JIRA and FishEye), Subversion, FTP and webmail restored to working order tomorrow, so please be patient – it will get there as soon as humanly possible.
The server is being physically relocated in a bit, as announced earlier. It should be back up again shortly…
The router has been fairly unstable over the last 5 days. It’s firmware has just been updated and everything reconfigured from scratch, but only time will tell if this has solved the problem. According to my ISP the old firmware did have a number of bugs in it – I just think it’s strange that these haven’t cropped up before, but there you go.
On another note, the site will be going down for maintenance right about now, but should be back up again within 15 minutes or so. I need to physically move the server, and thus have to disconnect it.
The server has been upgraded to Gentoo Hardended using Linux 2.6.11. All security measures (except ACLs) are in use.
Additionally, Apache now uses mod_security to sanitize requests, filter referer spam, and other niceties.
These changes may have broken some existing functionality. If you spot anything not working as before, please leave a comment or send us an email.
The software used to run the mertner.com blogs – WordPress – has been upgraded from an arbitrary daily build of v1.3 to the newly released v1.5. Getting the upgrade to work required changes in quite a few places, and chances are that something somewhere does not work quite right as a result of the upgrade.
If you find something of this kind, please send an email or add a comment to this post. (Assuming comment posting works, of course. If not, please yell loudly until we hear you or find the problem ourselves.)
We had another power outage yesterday at 22:13, causing a sudden reboot. It was repeated this morning at 4:14, and this time the server decided to halt with a BIOS message, leaving it unavailable until 13:01. Sigh!
Apologies for any inconvenience caused by these incidents.
A router was fried on sunday (20/3 at 9:30) in connection with a power outage, and could not be replaced until today. This caused mertner.com to be unavailable for a record period of roughly 50 hours, completely ruining any hopes of a nice uptime this year. The current uptime is around 99.2%, which is decent enough for a DSL-hosted site. But, it could be better ;-)