Version Date: 9-21-2022

Introduction

A goal of business planning is to mitigate disruption of product and services delivery to the greatest degree possible when disruption due to disaster occurs. Business continuity is the overarching concern.

An IT disaster recovery plan is the lynchpin of an overall business continuity strategy. And the purpose of business continuity is to maintain a minimum level of service while restoring the organization to business as usual. Contained within this document is the standard disaster recover procedures implemented by Laughing Rock Technology (LRT) on behalf of all qualifying customers. If a customer maintains an internal Disaster Recovery plan, it is to be used in conjunction with this plan.

Qualifying customers are generally defined as customers covered under a full-service contract at time of incident. If you are not sure if your contract falls under this policy, please see your LRT representative.

Versioning

To ensure you’re viewing the most up-to-date version of this plan, please refer to the Disaster Recovery Plan available on our website at: https://laughingrock.com/terms-and-conditions/

Information Technology Statement of Intent

This document delineates our policies and procedures for technology disaster recovery, as well as our process-level plans for recovering critical technology platforms and the telecommunications infrastructure. This document summarizes our recommended procedures. In the event of an actual emergency situation, modifications to this document may be made to ensure physical safety of our people, our systems, and our data.

Our mission is to ensure information system uptime, data integrity and availability, and business continuity.

Policy Statement

LRT management has approved the following policy statement:

  • The disaster recovery plan should cover all essential and critical infrastructure elements, systems and networks, in accordance with key business activities.
  • The disaster recovery plan should be periodically tested to ensure that it can be implemented in emergency situations and that the management and staff understand how it is to be executed.
    • For qualifying customers, backups are tested on a monthly basis to ensure backup integrity. These are random, partial file and system integrity tests. The information tested is at the discretion of the engineer performing the scheduled tests.
    • To ensure complete restoration in the event of a catastrophic event, full testing of restoration would be required. Full testing is performed only at the request of the customer and is treated as a project. Full virtual restorations are not covered as maintenance. To request a full restoration simulation, please contact your LRT representative.
  • The disaster recovery plan is to be kept up to date with mission critical applications and services. LRT depends on customer representative to ensure that we are aware of all mission critical applications.
  • A disaster situation is defined by LRT to be an event (or series of events) that results in one of the following scenarios:
    • Complete loss of access or functionality at any customer facility
    • A failure that makes a mission critical service or system unavailable to more than 50% of client staff simultaneously
    • A hardware failure occurs that impacts mission critical services or systems in any manner.
  • NOTE: It is the customer’s responsibility to maintain valid warranties on all mission critical hardware. The customer is also responsible for maintaining inventory of all mission critical replacement parts for hardware.
    • LRT may have temporary replacement hardware available in the event of a hardware failure. These include temporary servers, switches, firewalls, PCs, etc… Availability is not guaranteed and should not be relied upon as part of a recovery strategy. If temporary hardware is provided, it may incur additional fees.

Objectives

The principal objective of the disaster recovery program is to develop, test and document a well-structured and easily understood plan which will help the company recover as quickly and effectively as possible from an unforeseen disaster or emergency which interrupts information systems and business operations.

1 Plan Overview
1.1 Backup Strategy

In order to properly provide disaster recovery solutions, all customers covered under this policy must have the following solutions implemented.

  • For networks with onsite data storage, onsite physical servers, or onsite virtual servers:
    • Full backups via onsite appliance with offsite replication.
    • Backups are to be bare metal where applicable and configured to allow full system restoration even on dissimilar hardware.
    • Backups are to be run daily with a minimum of a 1-week retention period. Longer retention is strongly recommended based on backup storage availability.
    • Based on terms agreed upon at time of contract initiation, availability options may vary. Features such as local and cloud instant virtualization may not be available in all contracts. Please see your representative for details.
    • For networks without local server environments, cloud-only backup solutions may be used. Please see your contract terms for details.
    • Warranties on all mission critical hardware.
    • Replacement parts for all mission critical hardware.
  • For M365\O365 environments managed by LRT under contract, all SharePoint online storage to be backed up daily with unlimited retention.
    • LRT also recommends backing up all user mailboxes and OneDrive storage for users. Please see your contract to ensure coverage.
  • LRT DOES NOT backup 3rd party cloud applications. As these systems are frequently mission critical, please see your cloud application vendor for backup and retention terms.
  • For LRT cloud managed servers, a combination of VSS snapshots and daily cloud backups are implemented. Cloud backups utilize a standard 30-day retention policy.
    • NOTE: LRT cloud managed server policies are specific to Azure based server hosting. Other platforms will require custom solutions and are not defined in this policy.

1.2 Risk Management

This policy is designed to cover retention in the event of all manner of potential disasters. These disasters include (but are not limited to): Flood, Fire, Tornado, electrical storms, electrical outages, terrorism, sabotage, virus infection, user error, hardware failure, etc…

2 Emergency/Disaster Response Procedures and Objectives

2.1 Overview

2.1.1 Alert, escalation and plan invocation

All timelines are in business hours and begin at time or proper notification. Notification methods are described below. Escalation is determined based on effected systems and severity. This plan will be invoked at the discretion of the LRT team based in initial analysis.

2.1.2 Plan Triggering Events

There are multiple systems in place to trigger the activation of this plan including RMM alerts, automated system alerts, and customer notification via phone call. After LRT is notified, an initial assessment will be performed prior to officially invoking this plan.

2.2 Triggers and Notification

2.2.1 Key Triggers

Key trigger issues at headquarters that would lead to activation of the DRP are:

  • Total loss of all communications
  • Total loss of power
  • Flooding of the premises
  • Loss of the building
  • Loss of server reporting\contact
  • Hardware failure
  • Large scale loss of file\folder\internal data access
  • Carrier outage

2.2.2 Notification

When an incident occurs, the LRT Disaster Response Team (LRTDRT) must be notified. The LRTDRT will then decide the extent to which this plan must be invoked. Proper methods of notification are key to ensuring the correct personnel are notified in a timely manner. Please see the below for acceptable procedures to initiate a disaster recovery response via this plan:

Acceptable methods for reporting an emergency/disaster situation:

  • During LRT operating hours:
    • Call the LRT offices at 610-678-1978 and speak with a technician. If no one is available to answer your call, please follow the automated Emergency prompts to be routed to the Escalation Team.
  • After hours\nights and weekends:
    • Call the LRT offices at 610-678-1978 and follow the automated Emergency Prompts to be routed to an on-call technician

Unacceptable methods for reporting an emergency situation include:

  • Email: Our email-based support system is not designed to prioritize disaster messages over other emails. Do not email disaster reports to LRT.
  • Text Message: Similar to email, text messages through our system or to individual engineers are not designed to prompt an emergency response. Do not text message disaster reports to LRT.
  • Individual engineers\technicians: Our Disaster recovery team works on very specific protocols. Contacting individuals at any level of the organization directly may seriously delay our response time. Do not contact individual members of the LRT team directly to report a disaster situation.

2.3 Disaster Recovery Timeline Objectives

The team will be contacted and assembled once a disaster recovery event has been reported. The timelines set forth below do not initiate until actual contact has been made with an engineer and confirmation has been provided that the LRTDRT has been notified. The team’s objectives include:

  • Establish extent of disaster and required restoration strategies within 2.0 business hours;
  • Restore key services within 18-27 business hours of the incident being confirmed by LRTDRT;
  • Recover to business as usual within 18-45 hours after the incident being confirmed by LRTDRT;
  • Coordinate activities with client representative\primary client contact.
    • NOTE: The above are general timeline objectives. Actual timeline for specific recoveries will be dependent on the scope and extent of the disaster as well as the size of the recovery.

3 Insurance

As part of the company’s disaster recovery and business continuity strategies a number of customer held insurance policies may be relevant. These may include errors and omissions, directors & officers liability, general liability, and business interruption insurance.

If customer management feels that an event may be covered under insurance, LRT management should be engaged directly AFTER the incident has been resolved. The LRTDRT will assist in any relevant capacity to provide documentation and testimony for the claims process.