XMission Outage 11/11/08 – Updates and Moving Ahead

12 Nov, 2008 John W. News & Updates

Please note: As of January 1, 2017, XMission no longer sells DSL services.

We at XMission wanted to update our customers with more details and offer a proper apology now that the dust is clearing after yesterday’s outage, on Tuesday, November 11th.  This was a big one and all customers experienced problems to one degree or another.

This announcement is very long but we wanted to address questions and concerns that have arisen to restore customer confidence.

Synopsis

While the power only went out for a moment, many systems were adversely affected by the outage and took extensive attention and time to recover. In the case of our primary storage device, we could not bring it back online even after hours of trying so we restored files to the new NetApp we already had on site.  Many systems tie into this device, which exacerbated the problem.

We are happy to report that we will be greatly increasing base quota in the near future for customers at no additional charge from 100 MB to 5 GB now that we have the new NetApp storage appliance online. Our web hosting customers will also see significantly increased quota in the very near future.

As of the opening of business hours this morning, file backups had completed and most everything was in working order. We continue to find and address remaining issues, though. Some customers continue to experience delays with sending and receiving email but the queues are clearing.

Additional Technical Details

We didn’t have all the answers last night so here are further details regarding the outage:

• Our primary storage device (a NetApp F801 we were in the process of replacing this week with a new NetApp FAS2020) suffered the loss of 2 drives on one of the volumes, causing us to lose the data on the device entirely.
  • We were waiting to get our snapmirror license from NetApp to copy data over but at least we had the new hardware on site and ready.
  • Since many systems NFS mount to this gear, which handles /home, other servers required attention to get up and running properly.
  • Web hosting was down into the night until customer files were restored to the new hardware. This was completed by morning.
  • Our new NetApp 2020 has additional recovery options not available on the older 810.
  • We have plans to purchase another NetApp 2020 in the near future to host offsite. While we already have off site backups, the 2020 is an upgrade to that system.
• While email services were down about 5 hours for most customers yesterday, no mail should have been lost although some customers continue to see delays sending and receiving email.
• DSL and UTOPIA customers were offline for up to an hour because our radius server did not recover on its own. Some customers also needed to powercycle their modem before they could reconnect. As a rule of thumb we highly recommend customers powercycle their gear when troubleshooting.
• Although it was a holiday, our systems administrators were on site within minutes and many worked through the night, some up to 18 hours without a break.
• We are sorry about problems with our phone systems. They initially were offline due to the outage, then we maxed out connections to it, and we couldn’t answer the calls because we only had a skeleton staff of phone technicians due to the holiday. We are making some changes to the existing phone system but are also in the process of replacing it by the end of this year. Some expressed concerns that our status messages were not very helpful.  Unfortunately, we often did not know when systems would come back up and we also needed to keep the message short due to heavy call volume.
• To clarify, the outage was due to human error while doing maintenance on one of our 3 UPS’s and not any of our equipment. A breaker was mislabeled, which brought about the mistake.
• DNS (Domain Name Service) was sporadic for up to an hour. This was mostly due to a Cisco 6509 that continued to have issues in the beginning but we have since moved our two onsite authoritative nameservers (ns.xmission.com and ns1.xmission.com), as well as most other servers, to a new redundant connection.  We should note that we do have a tertiary name server in California (ns2.xmission.com).  If you list ns2.xmission.com as a tertiary nameserver for your domain, then your domain will continue to have working nameservice in the event that the two onsite nameservers are offline.
• QMOE customers suffered a prolonged outage due to the same Cisco 6509 that caused problems with our name servers. They have since been moved to a different router with greater redundancy. We will send our QMOE customers a separate email with further details by tomorrow.
• About 25% of our colocation customers suffered a brief power outage since we have customers spread across 3 separate UPS’s. Otherwise, aside from networking being briefly down after the initial outage, colocation services were not widely affected. In case some colocation customers are not aware, they can purchase powerstrips from different UPS’s to have redundant power. If you suffered equipment loss from the power outage or would like details about redundant power, please contact your sales rep for details.

Resolutions and Moving Ahead

We are very sorry for all of the problems that this outage has caused our customers and greatly appreciate all of the kind words and support you have given us. More than anything, we want to assure you that we are taking this matter seriously and proceeding with steps to lessen the chances that something like this can happen again, which include:

• We will more dutifully use our already existing NetStatus page to keep customers informed about our systems: http://stats.xmission.com/netstatus
• As well, we will be announcing all upcoming maintenance on the NetStatus page in the future and emailing those who opt into a list, which will soon be created for this purpose. To be added to the list, please email: support@xmission.com.
• We recognize that we need to handle communication much better in the future. We did setup an outage page with updates but realize that most did not know such a page existed: http://stats.xmission.com/outage/
• We also have our Nagios systems status page, which provides a very good look into our systems: http://stats.xmission.com/nagios/
• We plan to run redundant power from a second UPS up to our server room to feed essential hardware with dual power supplies. That alone would have dramatically minimized the effects of yesterday’s outage.
• While we already perform most maintenance outside of business hours, we have decided to enforce a policy that all systems critical maintenance (i.e., involving power, routers, core systems) must happen outside of business hours. Some additional training is also planned in regards to our electrical infrastructure.

news, outage, transparency, updates

XMission Outage 11/11 Ignite SLC