Component Seven: Test, and Test Often

Once you’ve implemented your disaster recovery solution you are on the right track towards positioning your credit union for success in the event of a disaster. However, IT is not static, it is ever changing. Therefore, it is important you implement the processes to adapt as circumstances change. Whether it’s an upgrade to existing servers, additional servers, changes to third-party vendors, or adding or removing branches, you need to have procedures in place that streamline the DR process.

On many occasions, during tests and even in actual disasters, we’ve encountered customers who have made changes and neglected to notify us. This oversight can be the difference in experiencing a successful disaster, and as I have already mentioned, you may only get one shot. By putting the proper procedures in place, you can avoid this by ensuring when you upgrade or add equipment and services, disaster recovery is always included on your check-off list.

It has become so commonplace that we implemented a process ourselves to send out a notification to our customers throughout the year, reminding them to review their DR services to ensure there have not been any changes that would affect their recovery. Additionally, we do a pre-DR test meeting to discuss and review both expectations for the upcoming test and their existing services to ensure nothing has been overlooked. Nobody wants to wing-it during a disaster. Preparation is key.

Now that you have clearly defined RTO’s and RPO’s you have a quantifiable way of measuring success during a test. Whether you are going with an in-house solution or outsourcing, you must ensure you keep track of your recovery times. This isn’t to say that by not meeting your RTO’s your test has failed. It simply allows you to determine if expectations are set too high and need adjusted, or if a different recovery method is required to meet the RTO.

As you proceed through a test, it’s critical you thoroughly document each individual process being recovered. Having detailed documentation allows for a post-DR test review to evaluate and determine if the recovery procedures were effective or if adjustments are needed.

Since a disaster is unpredictable, it’s important that you vary your disaster test scenarios. Many times, credit unions have a standard script they run through when performing their test. To be most effective, and increase your chances of success, you must enter each test simulating scenarios that are most likely to occur. For example, if you are located in the south east, you may want to simulate the effects of a hurricane or tornado. If you are in the Midwest, maybe a tornado or flood. And if in the west, maybe an earthquake or flood. However, don’t overlook a Murphy moment, a random scenario such as the failure of a single server or an irate employee wreaking havoc in the computer room.

Also, in large regional disasters, you may not have access to your entire IT staff. Even during hurricane Katrina many police and firemen did not show up to work because they were taking care of their family. This is a likely scenario, so you need to be prepared to function with minimal staff. If outsourcing, selecting a vendor that understands your business is a very real consideration and if you choose to go in-house, having adequate staff, cross-trained and working outside of the region will be required.

As you look at the various types of disasters it is clear to see there is a significant difference between a disaster where the data center is completely out of commission versus a single server recovery where routing of inter-dependent servers and third-party communications is needed.

With so many natural disasters making the news, disaster recovery has moved to the forefront for many auditors. However, this really isn’t as much about passing an audit as it is keeping your credit union in business and ensuring your members are able to access the funds they entrusted your credit union to protect. Without thorough testing on a re-occurring basis, you are playing a guessing game on whether or not you will be able to successfully recover from a bad situation.

So, there you have it! Disaster recovery is very complex. However, with the proper components in place, should the unforeseen occur, your credit union will be well prepared to experience a successful disaster.