Skip to content

Disaster scenarios

Purpose

This page describes the possible disaster scenarios that could be classified as P1/P2 Incident or Major Incident types and related action plans to address the issues in order to establish the platform recovery process. These scenarios are cross-referenced into the KaaS IT Recovery plan [F-00940499].

Related documents and Wiki page references

Failure modes and resolution plans

Possible failure modes are listed below.

1. Network disruption/ Connectivity issue - Priority: High - Failure Description: Network is owned and operated by GITO. KaaS team has no control over it. GITO network team is responsible for ensuring the availability of the network. - Resolution Steps: - [ ] Raise MI as per the MI process defined - [ ] Establish the communication channel - [ ] Refer Network Operation-Support for more details. A ticket can also be raised to ‘Network Operations (Corp)-L2-GL-IT’ to initiate the early CAPA process. - [ ] Verify the connectivity between CORP and KaaS - [ ] Resolve the Incident ticket - [ ] Notify the stakeholder

2. AWS Transit Gateway failure - Priority: High - Failure Description: Transit gateway is a network transit hub that is used to connect KaaS network to NN CORP network. This is currently owned by KaaS team and GITO is supporting to ensure the availability of Transit Gateway. - Resolution Steps: - [ ] Raise MI as per the MI process defined - [ ] Establish the communication channel - [ ] Refer Network Operation-Support for more details. A ticket can also be raised to ‘Cloud-AWS-L3-GL-IT’ to initiate the early CAPA process. - [ ] Verify the connectivity between CORP and KaaS - [ ] Resolve the Incident ticket - [ ] Notify the stakeholder

3. Disruption of ADO Services - Priority: Low - Failure Description: This failure mode could result in unavailability of KaaS configurations restored in KaaS ADO repos and possibly end up in unable to execute the pipelines. This failure will not impact the previously deployed configurations but certainly impact the new deployments. - Resolution Steps:
- [ ] Raise MI as per the MI process defined - [ ] Establish the communication channel - [ ] Connect with DevOps Platform Support for escalation and resolution. - [ ] Verify the pipeline functions by executing the pipeline IVs and/or Regression tests after discussion with Technical SME - [ ] Resolve the incident ticket - [ ] Notify the stakeholder

Note: Above specified resolution steps are followed when the impact is limited to KaaS ADO project. ADO services disruption at the organizational level is out of scope from KaaS disaster prospective as the impact and correction will occur at NN level.

4. Issues with Authentication Provider (Azure Entra) - Priority: High - Failure Description: This failure mode result in the loss of authentication and authorization (SSO) to confluent cloud setup and making the KaaS unavailable for actual users (Kafka clients connected via Service Accounts are not impacted). - Resolution Steps:
- [ ] Raise MI as per the MI process defined - [ ] Establish the communication channel - [ ] Verify the situation and accordingly use the emergency fallback user (NNG_UM_KAAS_emergency) account to re-establish the SSO setup. However, emergency fallback user utilization to bring back the SSO setup varies based on scenarios. - [ ] Resolve the incident ticket - [ ] Notify the stakeholder

5. Issues with AWS services (accounts enabling CI/CD) - Priority: Low - Failure Description: Issues with AWS services will result in disruption of CI/CD deployments of KaaS configurations into Confluent cloud from platform team and namespaces. This failure will not impact the previously deployed configurations but certainly impact the new deployments. - Resolution Steps: - [ ] Raise MI as per the MI process defined - [ ] Establish the communication channel - [ ] Refer Network Operation-Support for more details. A ticket can also be raised to ‘Cloud-AWS-L3-GL-IT’ to initiate the early CAPA process. - [ ] Verify the connectivity between CORP and KaaS - [ ] Resolve the Incident ticket - [ ] Notify the stakeholder

Platform Restoration plans

All source code of platform configuration is stored in Azure DevOps and can be used to re-establish the configuration of the KaaS platform during restoration.

The restoration steps are very much scenario based. A generic platform recovery process is illustrated as below.

KaaS Recovery process.png

IT Service Manager in consultation with Technical SMEs shall assess the requirement of platform recovery testing modes viz., redeployment through IVs and/or Regression testing or sanity checking and ensure the successful verification before resolving the Major incident.

Note: SITUATION MANAGEMENT TEAM can be approached for disaster recovery assistance based on the complexity of failure mode and multiple teams are required to be involved in resolving the MI.

Change Log

Version no. CHG no. Approved Pull Request Date Change Description
1.0 N/A PR192986 See PR New document
2.0 N/A PR193690 See PR Updated the Hyperlinks