{"id":36,"date":"2014-08-14T09:44:12","date_gmt":"2014-08-14T09:44:12","guid":{"rendered":"http:\/\/www.troliver.com\/?p=36"},"modified":"2014-08-14T09:44:12","modified_gmt":"2014-08-14T09:44:12","slug":"esx-clusterbomed-part-2","status":"publish","type":"post","link":"https:\/\/www.troliver.com\/?p=36","title":{"rendered":"ESX Clusterbomed &#8211; Part 2"},"content":{"rendered":"<p>Looks like the storage array is dead. After a bunch of messing around a couple of days ago, it really is apparent that we have lost the FFD2 enclosure.<\/p>\n<p>With it, we lose a few server, but we can also gain a load of storage disks. Until we manage come up with a new storage solution that has backups, I&#8217;m taking all the old SAS drives out to use as hotspares for FFD1. I have a feeling we have lots of 15k RPM SAS drives lying around that were used in other servers &#8211; more recently &#8211; and of the same brand to use to rebuild FFD2 again. It should work. If not, it&#8217;ll be a learning experience.<\/p>\n<p>I ended up familiarising myself a lot with the Dell Modular Storage Manager (DMSM) software and found out the correct way to assign hotspares and replace drives in the enclosure. A lot of messing around, unplugging and restarting took place on Tuesday, eventually resulting in a hot spare being designated as a physical replacement on another enclosure. I had actually written a good amount up about this but it was being written on notepad on a virtual machine that subsequently got restarted when &#8211; at some unknown point &#8211; it was decided to restart everything that was already running. Frustrating. But not the end of the world.<\/p>\n<p>Moving forward, what needs to be done now is:<\/p>\n<ul>\n<li>Have a backup solution:\n<ul>\n<li>If a server fails, if the hosts fail and if everything is lost, we need to be able to &#8211; at worse &#8211; rebuild and reconfigure those servers. Each server should have a designated backup plan associated with it<\/li>\n<li>Designate some replacement hot spare drives.<\/li>\n<li>Purchase a new storage array and an appropriate backup, with perhaps something like a scheduled daily backup of the system.<\/li>\n<li>Ideally the content from our internal wiki should be mirrored elsewhere so thatin the event of a disaster, we can recap on how to fix it.<\/li>\n<\/ul>\n<\/li>\n<li>Maintain the storage array and the ESX hosts more closely. Someone needs to monitor alarms as they appear and be informed of any storage array issues.\u00a0I also\u00a0need to look into why we no longer receive support emails automatically generated by alarms on the storage array (and this used to happen).<\/li>\n<li>Rebuild the vCentre server &#8211; probably on a physical host rather than a virtual one. Will need to look into that.<\/li>\n<\/ul>\n<p>For each of these points, I would probably make a new post &#8211; but this is just one part of what I am working with. FOG and the redeployment of our labs is also a priority, as are some other projects I have been working on lately. To be continued!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Looks like the storage array is dead. After a bunch of messing around a couple of days ago, it really is apparent that we have lost the FFD2 enclosure. With it, we lose a few server, but we can also gain a load of storage disks. Until we manage come up with a new storage [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[2],"tags":[12,18],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p6PQZ3-A","_links":{"self":[{"href":"https:\/\/www.troliver.com\/index.php?rest_route=\/wp\/v2\/posts\/36"}],"collection":[{"href":"https:\/\/www.troliver.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.troliver.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.troliver.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.troliver.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=36"}],"version-history":[{"count":0,"href":"https:\/\/www.troliver.com\/index.php?rest_route=\/wp\/v2\/posts\/36\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.troliver.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=36"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.troliver.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=36"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.troliver.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=36"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}