Disaster Recovery:Severe Weather Readiness

History

In the summer of 2024, two separate tornadoes missed Site:HME by a couple of miles. We were extremely lucky, however, shaken up by this incident. We determined there is nothing we could do about the weather but ensure our backups were up to snuff and plan in advance for severe weather. This guide was developed to ensure consistency and safety of operations going forward in the event of a hazardous weather situation.

Procedure

When it is determined there is a risk of severe weather, the following steps should be followed to prepare for the event:

1. Determine the risk of the severe weather.
   1. Local weather forecasts are used to determine risk and severity. MandolinSara constantly monitors College of DuPage Meteorology NexLab. This page is constantly updated every two minutes with with the latest severe weather warnings.
   2. To determine the chance of a tornado, we utilize [https://www.pivotalweather.com/model.php?m=hrrr&p=refcmp_uh001h PivotalWeather.
   3. If a significant risk event is imminent, proceed.
2. sysadminafterdark utilizes Monitoring:UptimeKuma to inform to public of outages, in addition to utilizing X, formally twitter. An incident should be raised on UptimeKuma and announced on X.
   1. UptimeKuma Procedure:
      1. Navigate to the UptimeKuma Dashboard. In the top, right hand corner of the screen, click "Status Pages", then click on the "sysadminafterdark" status page.
      2. In the top left corner of the screen, click the blue "Edit Status Page" button.
      3. Under the moon logo at the top of the screen, click the green "Create Incident" button.
      4. Change the style to "Info". The color should change to blue.
      5. Set the title to the following:

          Severe Weather Readiness - SiteID - Date 

      6. Set the Content to the following (Include formatting as markdown is supported):

         [Severe Weather Readiness](https://docs.sysadminafterdark.com/Disaster_Recovery:Severe_Weather_Readiness) has been enacted for site (siteID) for risk of (risk). Network downtime may occur due to this event. Please check @sysadafterdark and @mandolinsara X.com updates for further information. 

      7. Click the "Post" button.
   2. X.com Procedure:
      1. Navigate to X.com and create a new post with the following information:

          Severe Weather Readiness has been enacted for site (siteID) for risk of (risk). Network downtime may occur due to this event. Please check our status page for more information. https://status.sysadminafterdark.com 

      2. If downtime does occur, both sysadminafterdark and MandolinSara will be alerted via Monitoring:PagerDuty. A followup should be posted on X.com to detail the incident. Be as informative to the public as possible.
3. If it is safe to do so, the sysadmin should take a full backup of the environment with Disaster Recovery: Veeam Backup and Replication and push it to tape. If it is not safe, Pull the tape backup in the drive and shelter with it immediately!
   1. Procedure for taking a backup with Disaster Recovery: Veeam Backup and Replication can be found on this page.
   2. Once the tape backup has completed, another sequential tape should be placed in the tape deck. The most current backup should be kept on your person. Use the hard plastic covers located in Rack:HME-R02 to protect it.

Status

This action plan is standard operating procedure and has been enacted several times.