On top of all the other problems of the last week, my ISP had a network outage caused by a configuration error somewhere in their setup. Specifically, they have a circular fiber ring which somehow contrived to recycle traffic leading to a slow but inevitable meltdown of capacity.
As a result the server has been unreachable since roughly 01:00 CEST (GMT+1/UTC+2) up until around 23:00 CEST (~3 hours ago).
On behalf of my ISP, sorry for the inconvenience. At least it wasn’t me or my hardware this time :)
Getting Tomcat 5.5 to run on Ubuntu proved to be quite an exercise, as the stable package in Ubuntu Feisty was broken in multiple ways. I had to fix the logging setup (preventing Tomcat from starting properly), configure the proper JDK to use, add various required libraries needed for JIRA and Confluence, migrate applications and settings, and last, fix Ubuntu differences in the Apache proxy configuration. The good news is that JIRA, Confluence, Fisheye and Subversion now all are back online. The SQL backup was from 19/7 so anything added after that date will need to be re-entered.
There are still a great many things to be fixed, but at least all of the public services are available for use. Please let me know if you find anything that is broken, doesn’t work like it used to, behaves weirdly or any other suggestion relevant to the restoration process.
I have now completed restoring the most essential stuff on the new server, which is to say that web and mail is fully functional. Getting LDAP up and running proved slightly troublesome as the ldbm database had been deprecated and so Ubuntu had no support for reusing the backup files.
Next in line will be the secondary web apps – notably Tomcat for Jira and Confluence, Fisheye and RoundCube (for webmail). Why is it that Java apps are so brittle? It just never seems to work if you change anything even the slightest, unlike php or basically any other environment.
After that I need to rethink the backup strategy, set up mirroring on the secondary root disk, reinstall a virus scanner, configure the system logger properly, tune the display settings, and fix a whole slew of minor issues.
I’m hoping that at least the new distribution will prove to be stable; I’ve eliminated the Linux-VServer setup and consolidated everything on a single host. The new hardware should also result in significant speedups for everything – it has a quad-core 2.4GHz CPU and 4GB RAM with 2 (soon-to-be-mirrored) 10.000 RPM raptor disks. Perhaps even the Java apps will run satisfactorily on this ;-)
The recent instability problems seem to have been caused by the root disk in the server – as it has now failed completely.
I have purchased new hardware and am switching distribution (from Gentoo to Ubuntu) in order to avoid having to compile everything, and also because Gentoo has failed to deliver any kind of stability in recent times.
At this point I have several services up and running, most notably the Apache/MySQL combo enabling me to write this. Next up will be all the mail related services and dependencies thereof (such as the LDAP server), followed by the remaining web services (Tomcat, Subversion and Webmail).
I expect the remaining services to be more or less restored within the next 24 hours, but keep an eye out for updated status reports here.