The folks at Verizon apparently haven’t figured out how to create a redundant network. They have a piece of equipment identified as 0.so-7-0-0.XL1.MTL1.ALTER.NET which keeps crashing every 5 seconds, which effectively ruins any attempt at making a persistent connection between here and anything on the other side of that broken hardware item (such as, most of canada, and probably other parts of north america). Apparently they also don’t know of SNMP since it’s been doing it for hours and they still haven’t fixed it.
This just cost me a wad of money (hence the primary reason for me to rant about their blatant incompetence here). That said, if you live in north america and are unable to connect here – now you know why. Go call that 800-FCK-VRZN number!
Someone messed up and severed yet another fiber link this afternoon, causing the server to be offline from roughly 16:30. Repairs are underway and I expect the problem will be fixed shortly.
The server will be offline from september 9th 2006 20:00 until september 10th 12:00 (all times CEST = UTC+2).
The reason for this is that Global Connect needs to perform some major roadwork and repairs on the recently severed fiber links. Sorry all for the invonvenience.
Someone broke roughly 120 fiber cables this afternoon (at roughly 15:20), which has severed the link between my ISP and the world.
Global Connect (who owns the fiber cables) estimates that repairs will last until noon tomorrow (thursday), but as the fibers are repaired individually there is hope that the server will be reachable some time before that.
I’ll be using this opportunity to add some additional RAM to the server (from 2GB to 3GB), hoping that this will finally be enough even for the insatiable Java apps.
The server went down saturday at roughly 2:00 am and wasn’t rebooted until I came back sunday at 3:45 am. Unless these issues with 64-bit and/or Linux-VServer (still not too sure which of the two is the root cause) are resolved real soon, I’ll have to look at alternatives for the server OS. This is clearly not sustainable for much longer.. sigh!
It turns out my ISP was upgrading their network yesterday – replacing the core routers with new hardware as well as migrating to a redundant setup – but ran into severe problems, causing the main link to be unavailable for several hours (from roughly 19:00 yesterday up until this morning). Let’s hope they don’t have too many upgrades like that in store for us..
My primary ISP seems to have some kind of problem (my link has been cut a few times today, even if only for a few minutes each time). You can continue to access this site using the alternate link on http://mirror.mertner.com.
My ISP currently has a problem with a fiber link that affects 25% of their customers – including me. This means that the link to 18.104.22.168 is currently unavailable. Domains resolving to 22.214.171.124 (the DSL link to the server) can still be used (e.g. by visiting mirror.mertner.com instead of www.mertner.com).
The problem should be solved within an hour or two.
The server migration seems to be close to complete at this point.. FishEye is back online, and SpamAssassin is once again fully operational. I’ve also had time to track down and fix most of the known issues.. for instance, Tomcat would die for no reason with an OutOfMemory error (turns out it needed a larger PermSize, probably because it’s using a 64-bit JDK). I’ve also fixed a network card driver issue that caused the system to slow down quite a bit, so I am hoping that the system is stabilizing. FTP users have been given new passwords, so if you can’t login just send me an IM or email.
There are still a few issues to be fixed, so should it happen that a service is down for a minute, please bear with me. If anything is down for more than a few minutes, feel free to start complaining again ;-)
JIRA and Confluence are back online, and I’ve restored the missing plugins and corrected a few minor issues. FishEye should be back tomorrow too.
Subversion is also back, but the Gentle repository has a new URL (it is now http://www.mertner.com/svn/gentle instead of http://www.mertner.com/svn/repos). Note that the “projects” folder has been stripped on import. This means that you will have to relocate your working copy to the new URL. If you previously checked out /svn/repos/projects/gentle then you should relocate this to /svn/gentle/gentle (I tried to strip out the extra gentle too, but then Subversion would not be able to import it for some reason). The NProf repository has not moved and is still available at /svn/nprof.
Apparently the PHP developers have decided that using recode along with mysql is a bad thing, which has broken international characters on some domains. I’ve tried convincing PHP of compiling in support for both anyway, since it used to work just fine, but alas, it just wont no matter what.