{"id":325,"date":"2008-11-12T10:36:04","date_gmt":"2008-11-12T16:36:04","guid":{"rendered":"http:\/\/xmission.com\/blog\/?p=325"},"modified":"2014-02-14T14:18:51","modified_gmt":"2014-02-14T21:18:51","slug":"xmission-outage-1110","status":"publish","type":"post","link":"https:\/\/xmission.com\/blog\/2008\/11\/12\/xmission-outage-1110","title":{"rendered":"XMission Outage 11\/11"},"content":{"rendered":"

Hello Everyone,<\/p>\n

XMission extends a heartfelt apology to our customers for any inconvenience caused by yesterday’s outage. We are truly sorry, and appreciate your patience.<\/p>\n

This is a courtesy post for anyone that is not already on the XMission Announcement email list.<\/p>\n

XMission Outage<\/span><\/h2>\n

XMission experienced a serious outage while we were performing some standard UPS maintenance today<\/span><\/span>. The outage affected all services and started at approximately 2:00 p.m. on Tuesday<\/span><\/span>, November 11th. Network services for many were partially restored by about 2:30 p.m. but some other services required a lot of attention and took much longer.<\/p>\n

Details<\/span><\/h2>\n

About 40% of our data center, including our server room, suffered a power outage when a technician flipped a mislabeled breaker during some standard maintenance on one of our 3 UPS units. Although the power outage was momentary, servers and routers often respond very poorly to losing power and sometimes take extensive work to come back up. Unfortunately, such was the case today<\/span><\/span> with many systems.<\/p>\n

Seriously Affected Systems<\/span><\/h3>\n
    \n
  • An important router, which some connections and servers rely on, required extensive attention from our network administrators.<\/li>\n
  • DNS (Domain Name Service) was sporadic for some customers for over an hour.<\/li>\n
  • Email services were down for over 5 hours.<\/li>\n
  • Web hosting suffered the longest outage because our NetApp storage appliance which houses all customer files and web sites lost multiple hard drives. As a result, we are currently restoring files to our new NetApp 2020 from our November 9th backup, which will take many hours yet to complete. We recently purchased this new NetApp and were merely days away from getting it online.<\/li>\n<\/ul>\n

    Conclusion<\/span><\/h2>\n

    Today<\/span>‘s outage was exacerbated by multiple systems responding poorly to losing power. In spite of the holiday, our systems administrators were on site within minutes and continue to work tirelessly to restore all services. In the end, we should have performed this maintenance on a day when our systems administrators were on site because problems can arise no matter how carefully you proceed.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Hello Everyone, XMission extends a heartfelt apology to our customers for any inconvenience caused by yesterday’s outage. We are truly sorry, and appreciate your patience. This is a courtesy post for anyone that is not already on the XMission Announcement email list. XMission Outage XMission experienced a serious outage while we were performing some standard […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[291,293,292],"class_list":["post-325","post","type-post","status-publish","format-standard","hentry","category-news-updates","tag-news","tag-outage","tag-updates"],"_links":{"self":[{"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/posts\/325","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/comments?post=325"}],"version-history":[{"count":16,"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/posts\/325\/revisions"}],"predecessor-version":[{"id":4718,"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/posts\/325\/revisions\/4718"}],"wp:attachment":[{"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/media?parent=325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/categories?post=325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/xmission.com\/blog\/wp-json\/wp\/v2\/tags?post=325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}