The folks at EngineHosting had a spot of hardware trouble earlier this evening that knocked a whole bunch of us off the net temporarily. Here’s part of the email I received about it.
Sorry folks, we have a number of back-end ethernet switches that handle database, storage array, and other important communications between our clustered servers. We quickly identified that it was indeed an issue with our backend network, but as all visible indicators showed no outward issues (ie: no dead switches, typical ethernet switch blinking lights, and no warning/alert lights were on). Deeper investigation identified a highly unusual issue between two switches sharing a high-speed uplink port to pass data between them, we found the switch with the issue pulled it out of service and replaced it with a stand-by switch and systems are again operational.
They’re taking steps to upgrade hardware and put a new ticket system in place to help avoid outages like this again so I’m happy. In fact I believe this is the first ever unscheduled downtime SEB has suffered since we switched to EngineHosting. Can’t complain too much about that.