Protect Your Business by Implementing an IT Disaster Recovery (DR) Plan

Disasters can strike any business without warning. That’s just a fact of life.

The good news is that you can take steps to be prepared. By doing so, you can ensure the effect on your customers and employees is minimal or at least manageable.

In the modern age, businesses depend on technology for their day-to-day operations. A disaster resulting in even a short-term loss of IT resources could have momentous results on the bottom line and customer loyalty.

A good IT Disaster Recovery (DR) Plan is essential to ensure business success under virtually any eventuality.

Why Do I Need An IT Disaster Recovery Plan?

One quick look at the evening news should tell you that the threats to businesses are real. Everything from hurricanes and earthquakes to security breaches and malicious employees can affect the ability of your IT department to support your operations.

Even short outages are expensive. To begin with, there are the direct costs associated with recovering from the disaster. These include technical personnel, repairing or replacing equipment, and so forth. Additionally, during a disaster you may lose business, need to pay employees over time, hire additional help, and face public relations problems.

Don’t think for a moment that you’ll be off the hook regarding the compliance requirements of your industry (i.e., PCI, HIPPA, SOX, TJC etc.) in the event of an emergency. You are required to maintain the same levels of privacy and security even under the worst-case scenario.

Cloud-based IT functions might lead you to believe that you are safe from disasters. But have you considered the paper files, mobile devices, laptops and desktops, and even Rolodex files that are in your offices? What about the people who run your business? These concerns lead to a wider discussion regarding business continuity planning (BCP) which involves all facets of your organization.

What Types Of Disasters Should I Prepare For?

It can be overwhelming to think about all the kinds of disasters and failures that can occur in the IT area. Begin by asking yourself what events are likely in your city and state. For instance, in California, earthquakes are a possibility; on the eastern seaboard, hurricanes; and for the Midwest the concern is tornadoes.

Often overlooked are local hazards such as paper mills (highly flammable), chemical plants and storage facilities (poison gas and explosives), gas lines (can rupture) and so on.

Keep in mind, the most common disaster scenarios are small-scale but significant to the business. Equipment failures, water leakage onto IT resources, pest infestations and so forth. For instance, a power failure or disk corruption is far more likely than any natural disaster. Don’t overlook these.

Security penetrations, data corruptions, ransomware, viruses, malicious employees and so on can render your IT resources unusable just as quickly and with more impact than a fire or explosion. These less obvious dangers must be included in your IT DR planning.

Finally, as the cloud becomes more integrated with your business the effects of communications failures become increasingly significant. The loss of internet connectivity can put your office, warehouses or stores out of commission as thoroughly as any major disaster.

Support From the Business

Without support of senior leadership in the business, your IT disaster recovery plan won’t go very far.

Get the primary stakeholders involved early. Support must come from C-suite executives or at least department heads across the business. These leaders must be involved at a high-level in every stage in your planning. Gain their support by stressing the importance of keeping their areas operational during disasters.

Their agreement and support are critically important. Without it, you’ll find it difficult to get funding, involve their teams in planning and testing, and to be successful when a disaster does occur.

How to Write a Disaster Recovery Plan

Begin by working through a recovery plan for each of the primary disaster scenarios. Ask yourself “what if” questions. What if there is a hurricane? What if a cyberattack corrupts your databases? What if one of your warehouses has a fire that puts it out of commission?

Now, perform a business impact analysis (BIA). What do each of these scenarios mean to the business, employees and customers? Staff might be evacuated because of a hurricane, your point-of-sale system could be down if the databases were corrupted and shipments from or to the warehouse could be halted because of the fire.  In other words, evaluate the effects of these different disaster scenarios for financial, legal, safety, and other impacts on the business.

Once you’ve completed this analysis you can begin the planning and implementation of your DR solutions.

What are the Key Elements of a Disaster Recovery Plan?

Briefly, the key elements of a successful disaster recovery plan are summarized below.

1. Know the hazards. As mentioned earlier, understand what can go wrong and how that effects the business.
2. Understand applications, databases and data flows. Know how applications interact, where data is stored and what accesses that information. Look for interdependencies. Create a master data flow diagram that drills down to individual applications and keep it updated. Begin by focusing on mission critical operations.
3. Decide on your DR solution. Are you implementing a full hot site or, at the other extreme, just duplicating your data to another location? Many organizations choose a combination of several solutions, using a hot site for mission critical applications while just replicating data for less vital ones. In concert with your stakeholders, determine the cost and benefit for each proposed solution.
4. Partial failovers. Plan for partial as well as full-scale disasters. Sometimes only a single server, computer room or database may be affected by the scope of the event. Ensure you plan for these smaller issues as well.
5. Write up the standard operating procedures to following during the disaster. Many plans forget this vital fact – during and after a disaster, some or all your IT functions may be running out of your DR location. Make sure you include the SOP for this contingency as well as SOPs for failing over and restoring back to primary systems.
6. Test your assumptions. As you work through your plan, write down every assumption. Test each one of these. There is little that can sabotage a disaster situation faster than an incorrect assumption.
7. The plan must be in writing. If it is not written, it isn’t going to work. Keep the written plan up to date as the applications, systems and the business change.
8. Remember people. Do you honestly believe that your IT team will come into the office after an earthquake? Include alternatives to job roles for each critical team member – and don’t forget about those outside of IT. Your stakeholders can help identify those people.
9. Perform tabletop exercises regularly. A tabletop exercise is intended to examine the responses of your crisis team during a disaster scenario. Their purpose is to help you identify gaps in your plan. Include a few stakeholders if possible so they can contribute their expertise about their area of responsibility.
10. How do you perform a tabletop exercise? Decide and plan out a scenario. Assign a moderator who walks the team through the scenario step-by-step in real-time. Each member of the exercise performs and discusses the actions they would take. After the exercise is complete, debrief the team, analyze the results and implement changes as appropriate. Note that FEMA offers a free course called Incident Command System 100 and National Incident Management Systems 700A-level training about this subject.
11. Test your plan periodically. Any plan, no matter how good, will quickly become useless without testing. In addition to tabletop exercises, perform actual failover (and failback) testing at least on an annual basis.
12. Don’t forget about compliance. Security, privacy and other compliance standards are not relaxed in the event of a disaster of any magnitude. Ensure your applications always remain compliant.
13. The Cloud doesn’t obviate disaster recovery planning. Just because your resources and applications are in the cloud doesn’t mean you don’t need to worry about DR. Always carefully examine your cloud and service provider SLAs and contracts. Ensure those agreements adequately cover disaster scenarios, timings, and failure modes.
14. Consider business continuity. IT doesn’t work in a vacuum. The business must be involved in the entire plan. Even if your IT DR plan works perfectly, the business may fail if they are not prepared.

Conclusions

Disaster recovery and business continuity plans are insurance policies. The business stakeholders must weigh the risks versus the benefits of its DR solutions, decide which options are optimal for their business, and then implement accordingly.

For some businesses, simply replicating the data to a remote site every evening is sufficient. Others, especially those that directly serve the public or manage critical infrastructure, may need high availability and fully redundant sites.

Your IT group and your company can weather disasters of virtually any magnitude by taking the time to analyze, plan, implement, and maintain your DR and BC plans.  It’s in your hands now…best wishes!