The digital infrastructure sector is struggling to achieve a measurable reduction in outage rates and severity even though downtime at data centers can result in significant economic costs and problems for organizations, including lost revenue, damage to reputation, and increased expenses related to recovery efforts.
“Digital infrastructure operators are still struggling to meet the high standards that customers expect and service level agreements demand – despite improving technologies and the industry’s strong investment in resiliency and downtime prevention,” said Andy Lawrence, founding member, and executive director, Uptime Institute Intelligence. “The lack of improvement in overall outage rates is partly the result of the immensity of recent investment in digital infrastructure, and all the associated complexity that operators face as they transition to hybrid, distributed architectures.”
The Uptime Institute’s 2022 annual Outage Analysis report found that downtime costs and consequences were worsening, making the management of data center connectivity a linchpin of both business continuity and disaster recovery plans.
Why Data Centers Experience Downtime
Here are some examples of what can cause downtime at data centers:
- Hardware or software failure: Hardware components like servers, storage devices, and networking equipment can fail due to age, wear and tear, power outages, or environmental factors such as temperature and humidity. Similarly, software systems can experience glitches or crashes that can bring down critical systems and applications.
- Human error: Misconfigurations, accidental deletions, and other human errors can also cause downtime. For example, a system administrator might accidentally delete an important file or make a configuration change that causes a system to fail.
- Cyber-attacks: Data centers are prime targets for cyber-attacks, and a successful attack can lead to significant downtime. Malware, ransomware, denial-of-service attacks, and other types of cyber-attacks can compromise systems, steal data, and cause systems to fail.
- Natural disasters: Natural disasters like earthquakes, hurricanes, and floods can also cause downtime at data centers. These events can damage data center infrastructure and disrupt power and communications networks.
Not only can the economic costs of downtime at data centers add up quickly – in terms of lost revenue and productivity, as well as recovery effort costs – but downtime can damage an organization's reputation and lead to the loss of customers and business opportunities.
Customers may become frustrated or lose trust in the organization if their services are unavailable or if their data is compromised. This can result in a loss of business and damage to the organization's brand.
Data Center Downtime Costs and Consequences Worsening
Business continuity plans (BCP) and disaster recovery plans (DRP) are more important than ever, especially given the key findings from the2022 annual Outage Analysis:
- High outage rates haven’t changed significantly. One in five organizations report experiencing a “serious” or “severe” outage (involving significant financial losses, reputational damage, compliance breaches, and in some severe cases, loss of life) in the past three years, marking a slight upward trend in the prevalence of major outages
- The proportion of outages costing over $100,000 has soared in recent years. Over 60% of failures result in at least $100,000 in total losses, up substantially from 39 percent in 2019. The share of outages that cost upwards of $1 million increased from 11 percent to 15 percent over that same period.
- Power-related problems continue to dog data center operators. Power-related outages account for 43 percent of outages that are classified as significant (causing downtime and financial loss). The single biggest cause of power incidents is uninterruptible power supply (UPS) failures.
- Networking issues are causing a large portion of IT outages. According to Uptime’s 2022 Data Center Resiliency Survey, networking-related problems have been the single biggest cause of all IT service downtime incidents – regardless of severity – over the past three years. Outages attributed to software, network and systems issues are on the rise due to complexities from the increasing use of cloud technologies, software-defined architectures, and hybrid, distributed architectures.
- The overwhelming majority of human error-related outages involve ignored or inadequate procedures. Nearly 40 percent of organizations have suffered a major outage caused by human error over the past three years. Of these incidents, 85 percent stem from staff failing to follow procedures or from flaws in the processes and procedures themselves.
Preparing for the Unexpected: Business Continuity vs. Disaster Recovery
BCPs and DRPs are both essential components of an organization's overall strategy for managing disruptions to business operations:
- A BCP is a comprehensive plan that outlines how an organization will continue to operate during and after a disruption. It focuses on the overall continuity of business operations and takes a holistic approach to identifying and mitigating risks. A BCP typically includes strategies for maintaining critical business functions, communication plans, and protocols for activating and managing the plan.
- A DRP, on the other hand, is a more specific plan that outlines how an organization will recover IT systems and infrastructure after a disruption. It focuses on restoring IT systems and data to ensure that critical business functions can resume as quickly as possible. A DRP typically includes strategies for backing up data, testing backup systems, and restoring systems and data in the event of a disruption.
The two plans are alike in that they both aim to ensure the continuity of critical business functions and minimize the impact of disruptions. They also share some common elements, such as risk assessments, testing and training, and communication plans.
However, there are some key differences between the two. While a BCP takes a broad view of business operations, a DRP focuses specifically on IT systems and infrastructure. Additionally, a BCP is focused on maintaining operations during a disruption, while a DRP is focused on restoring systems and data after a disruption.
Organizations should have both BCPs and DRPs in place to ensure that they are prepared for all types of disruptions. Together, these plans can help organizations ensure that critical business functions can continue in the event of a disruption and that IT systems can be quickly restored to minimize the impact on business operations.
Maintaining Connectivity Key to Both BCPs and DRPs
In a data center environment, ensuring connectivity is crucial for implementing both business continuity and disaster recovery plans.
Here are some best practices for managing data center connectivity to ensure both succeed:
- Establish Redundant Connectivity: Having multiple connections from the data center to the Internet is essential for ensuring connectivity in the event of a disruption or disaster. Redundant connectivity could include dual Internet Service Providers (ISPs), multiple circuits from different carriers, or even a mix of wired and wireless connections. The goal is to ensure that if one connection fails, there is another to take its place.
- Implement Load Balancing: Load balancing across redundant connections can provide an additional level of redundancy and ensure that traffic is distributed across multiple paths. This helps to prevent a single connection from becoming overloaded, which could lead to poor performance or an outage.
- Use Virtual Private Networks (VPNs): VPNs can provide secure connectivity to remote workers and other locations, including backup data centers. VPNs encrypt data in transit, which helps to protect against data breaches and unauthorized access.
- Maintain Accurate Network Documentation: Accurate documentation is essential for managing data center connectivity. This includes detailed diagrams of the network topology, IP address assignments, circuit information, and contact information for key personnel. Having accurate documentation can help to speed up the recovery process in the event of an outage or disaster.
- Regularly Test and Update the BCP and DRP: A BCP and DRP are only effective if they are regularly tested and updated. Regular testing helps to identify weaknesses in the plans and provides an opportunity to improve them. Updating the plans as the network infrastructure changes is also critical to ensure that they remain relevant and effective.
- Plan for Power Outages: Power outages are a common cause of data center disruptions and disasters. Ensure that your BCP and DRP include plans for backup power sources such as generators or uninterruptible power supplies (UPS). Also, make sure that critical systems are connected to these backup power sources.
- Replicate Data: Replicating data across multiple data centers can provide an additional layer of redundancy and ensure that critical data is not lost in the event of a disaster. Data replication can be achieved through technologies such as data mirroring, data backup, and data synchronization.
- Train Staff: Ensure that staff are trained on the BCP and DRP and their roles and responsibilities in the event of a disruption or disaster. Regular training sessions can help to reinforce the plans and ensure that staff are prepared to respond appropriately.
By implementing these best practices, organizations can improve their data center connectivity management to ensure both business continuity and disaster recovery in the face of unexpected disruptions and disasters.
How Colocation Can Help Data Centers BCP and DRP
Colocation can help data centers maintain connectivity in business continuity and disaster recovery situations in several ways:
- Geographic Diversity: Colocation providers often have multiple data centers located in different geographic regions. This provides businesses with the opportunity to select a colocation provider that has data centers in different locations, reducing the risk of a single point of failure and increasing the chances of maintaining connectivity during a disaster.
- Redundant Connectivity: Colocation providers often have multiple connections to multiple Internet Service Providers (ISPs). This provides businesses with redundant connectivity to the Internet, ensuring that they can maintain connectivity even if one ISP experiences an outage.
- Backup Power: Colocation providers typically have backup power sources such as generators and uninterruptible power supplies (UPS) to ensure that their data centers remain operational during power outages. This reduces the risk of downtime and ensures that businesses can continue to maintain connectivity during a disaster.
- Physical Security: Colocation providers typically have advanced physical security measures such as biometric authentication, surveillance cameras, and security personnel. This provides an added layer of protection to data centers and ensures that businesses can maintain connectivity even during a physical security threat.
- Expertise and Support: Colocation providers typically have experienced personnel who can provide support and guidance during a disaster. This can be invaluable to businesses that may not have the same level of expertise and resources to manage a disaster.
Colocation can help businesses minimize the impact of unexpected disruptions and disasters on their operations and maintain connectivity during critical times.
