Disaster Recovery Planning
The best time to plan for a disaster is before it arrives. Unfortunately, this advice does not help in recovering from major damage. Fortunately, with today’s energy management solutions, equipment can be installed and prepared before and after a catastrophic incident. By ensuring commercial facilities are retrofitted with the latest energy-efficiency lighting and control systems, businesses and governments can ensure that vital operations data is preserved and operations can come back online as soon as possible. Smart systems return online and recover with little need for human intervention. As a result, businesses do not need to wait days or weeks for emergency rescue response to help mitigate damage and recover operations. Labor costs for maintenance and troubleshooting are also significantly reduced. Such reliability is essential to commercial businesses and institutions surviving a storm.
There are no one-size-fits-all templates for disaster planning. Various systems within a facility respond differently to loss of power. After a disaster, power should be restored to the most critical services first, but the definition of “critical” changes depending on the duration of the outage. Freezers and refrigerated storage may be critical systems, but once the contents reach a critical temperature and the contents are lost, other loads within a facility may become more critical.
Attempting to sort out these priorities during the chaos that follows an event makes decision making more difficult. If a disaster should occur, the consequences of electrical power loss can be minimized with established emergency procedures.
Train employees so they know what to do. Make emergency preparedness part of the culture of an organization.
Before Disaster
- Obtain a qualified first-response service provider with the breadth and depth of trained and experienced personnel for the equipment at your facility.
- Spend the time to identify and meet with those resources that could be contacted for disaster support. Research the providers’ capabilities. Make sure its personnel have toured the facility and have identified critical areas. Recognize that for widespread disasters like hurricanes, employees will be affected. Make sure the provider can source people and materials from out of the area as required.
- Perform a pre-crisis risk mitigation audit to estimate the potential impact of credible disaster scenarios and identify ways of minimizing vulnerability in the event of a disaster.
- Perform a critical load audit to identify all loads that require backup power (which may be more than what is actually backed up today).
- Identify consequences of potential natural (e.g., flood, tornado, hurricane, earthquake) and man-made (e.g., terrorist, human error within organization) threats. This analysis should take into account physical surroundings (e.g., proximity to rail, airports, ship ports, or highways) and includes financial impact resulting from the loss of that equipment.
- Outline consequences of electricity loss (e.g., computer failure, loss of access, contamination, trapped persons, chemical release, etc.) with varying durations of outage. Have a contingency plan to deal with each consequence (e.g., manual key entry backup to electronic locks).
Note: Pre-crisis audits provide the additional benefit of potentially identifying internal problems—remember, not all problems are caused by external events—that could cause several issues including: storage blocking equipment access/escape routes, missing breaker racking or lifting tools, missing drawings, etc.
- Conduct a safety audit and establish procedures to assure injury free remediation.
- Assure regulatory compliance awareness
- Identify all critical documentation and create a plan to store this information so that it can be accessed off-site in one or more safe locations. Do not assume that cloud communications will be available in all disaster scenarios.
- Consider adding local electrical power generation. This can take the form of:
- Permanent onsite local generation is high cost, but eliminates the need to rely on a rental company having a generator when you need it. However, this increases responsibility to ensure the system is functioning. A common solution is to contract out maintenance to a qualified engine dealer, typically the seller. Don’t forget to contract with fuel providers for fuel delivery since normal fuel delivery will likely be affected after a problem that strikes a wide area.
- Add provisions for temporary power hook up. Opening and working on electrical equipment can only be done by trained, certified professionals, and this includes connecting an emergency generator. Remember, during times of emergency, trades people will be in high demand and short supply as the hard work of restoring power begins. Consider having provisions already installed to allow simply plugging-in a backup generator.
- Alternative energy sources, such as a solar energy system, are typically designed to operate when utility grid power is available and automatically shut down when utility power is lost; however, in times of emergency, having electrical power is highly desirable. Meet with a solar installation provider or a qualified engineering service provider to explore ways of configuring local alternative energy sources into an “island” or a microgrid on an as-needed basis. This may involve adding additional protective devices or devices to automatically shed lower priority loads since alternative energy is rarely sized to power an entire building’s load.
- Identify sources of equipment repair and replacement. These sources should be certified for the equipment installed. Since many facilities are older and may include electrical equipment from a variety of electrical vendors, look for sources that have the certification or other demonstrated proficiency to repair, renovate, and renew the electrical equipment installed at your facility.
- Make sure contracted support organizations have expertise in staging of support equipment including generators, replacement electrical equipment, and satellite communication networks.
- Develop a plan for living and support accommodations for an in-house crisis response team, and ensure that contracted support teams are similarly prepared. Food, water, and sleeping accommodations may be in short supply so make sure support teams can support themselves.
- With a radio communication system, ensure its operability following an electrical system failure. Typically this involves supplying power to chargers and repeaters.
- Document your equipment installed (brand, model, serial), device settings, and software (vendor provided and user purchased). Update documentation when you buy or change equipment or settings. Make sure new staff members are trained on this procedure as previous staff leave. Have clear responsibilities as to who is responsible to keep the data updated.
Recover after Disaster
- Depending on how widespread the disaster, recognize that failure of communication systems may prevent contacting service providers. Consider prearranging with those providers to check-in following a wide area event.
- Execute disaster recovery plan. Mobilize disaster recovery team, each with assigned tasks.
- Team will include internal company personnel and external contractors.
- Use established team leaders to prioritize tasks. Consider outsourcing project management for specialty items such as electrical equipment repair or restoration.
- Flood waters conduct electricity. Entering a flooded building, especially rooms containing electrical equipment, is dangerous. Only trained personnel skilled in operating in this environment should enter such a facility.
- Whether or not you have done any preplanning, once a disaster hits, don’t panic. Choose electrical service providers carefully. While qualified service personnel may be in short supply following a wide area disaster, be careful in your rush to find support. Hire only qualified service providers. Several manufacturers of electrical equipment have programs where electrical service providers are certified by the manufacturer. Review their recommendations by visiting manufacturers’ websites. A listing of electrical equipment manufacturers is available at www.nema.org/mfgs.
- Once you have recovered from a disaster, spend time reviewing what worked, what didn’t, and what could have been done differently or prevented. Update your plan as equipment is bought, upgraded, changed, or repaired.
Example
Hurricane Rita, one of the most intense Atlantic hurricanes recorded, struck Lake Charles, Louisiana, on September 24, 2005. A chemical manufacturing plant in Lake Charles found itself in dire circumstances as Hurricane Rita left the facility without power at a critical point in its production cycle. Several million dollars of process materials and equipment were at risk and would result in a total loss within one week without restoration of power. By using an outside team skilled in this type of emergency recovery, the equipment was saved and shipping deadlines met.
Roadmap Recommendations
- Develop relationship with a proven service supplier.
- Review and understand critical power points within the operation.
- Analyze opportunities for alternative power and backup generation for critical power loads.
- Evaluate the energy management system that is right for your facility. Lighting and energy solutions companies can evaluate the size of the facility, nature of business and operations, disaster threats (e.g., earthquakes or flooding), and other considerations to recommend and quote newer, smarter technologies suited to specific needs.
- When a new system is installed, ensure it is properly programmed for emergencies. If possible, connect the system to the building automation system emergency protocols. Run diagnostic tests to ensure that lighting and data recovers in the event of emergencies.
- If you have experienced storm damage, explore retrofit options. For example, energy-harvesting wireless solutions are ideal for replacing and retrofitting damaged buildings. They can be placed and programmed in minutes, and do not require pulling wires. As a result, essential business functions can continue uninterrupted, expediting recovery.
- Investigate upgrades for existing systems. If you have already invested in a distributed control or centralized control system, investigate what options are available to upgrade to newer, smarter technologies with reliable recovery and constant data preservation. These improved technologies are safer with higher voltage ratings, and much speedier, smoother recovery times after power interruptions.
Reference