Standard Changes¶

Purpose¶

This page describes how standard changes are handled on KaaS. Standard changes are changes where the impact and risk are well understood (typically low complexity configuration changes), and in KaaS we use pre-designed ServiceNow templates to accelerate the processing. No manual, formal testing is required with these types of changes.

Prerequisites¶

In order to create Standard Change templates in ServiceNow, you must have the "ServiceNow ITSM : ITSM Change Manager (License Cost Applicable)", which can be requested in novoAccess. The Templates should be reviewed by the KaaS Service Manager and Quality Controller before being made available for general use.

Process Responsibility¶

It is the responsibility of the KaaS Service Manager to process the standard changes; initiate and ensure closure with adequate references in ServiceNow.

KaaS Standard Change Templates¶

Standard Change can be selected in ServiceNow when creating a new change. These are the relevant templates for KaaS (continuously updated as more are added).

Pipeline Agent Update¶

This standard change enables an image update of the Pipeline Agent component in KaaS. This component is required for LOB pipeline connectivity into KaaS, so that projects/teams can deploy their data products into KaaS (private network) from their own ADO project. It is important to note, that the component does not directly influence any data or configuration on the platform - only connectivity to pipelines, who can deploy configuration. This means, that in a worst-case scenario, LOB teams will be unable to deploy new configuration to the platform until the Agent is functional again, but existing data products are not affected at runtime.

To mitigate human errors and reduce maintenance of the component, automation has been implemented. More details can be found in GIT repo here Link. This includes a security scan of the 'stable image' to check for vulnerabilities, and an automatic build of an updated 'patch image', which can then be approved/promoted to 'stable image' by a manual approval. This Standard Change will be referenced when approving.

Change Handling & Operational Practicalities¶

These instructions guide the handling of the Pipeline Agent update change; - Platform Operators or Platform Architect can approve the promotion from patch to stable. A reference to the ServiceNow standard CHG ID must be made when approving. - In case of issues with the image, an older version can be retrieved and applied. More details and guidance can be found in the AWS-Infra: Pipeline Agent: Operation Guide - Configuration related to the product is stored in ADO and AWS, where it is version controlled and access is managed with policies and roles. There is no dedicated CIL. - Significant refactoring and new feature introduction (e.g. more automated testing) needs to be assessed, reviewed and approved by the Platform Architect. Assessed risk and impact may lead to a dedicated CHG being created.

Template Link: Pipeline Agent Update

Team Onboarding¶

This standard change enables the creation and onboarding of a new 'Team' ("namespace") and its associated resources and permissions. This includes capturing details on owners/approvers in an onboarding agreement, User Access Management (AD group creation, novoAccess onboarding), Data Product Pipeline Template enablement.

Change Handling & Operational Practicalities¶

These instructions guide the handling of Team Onboarding change; - Before any requests for creation are made, the Onboarding Agreement must be filled out and signed. This should be handled by Service Manager or delegate with a Team/Namespace representative. - Detailed guide and process for requesting are found at Team Overview & Onboarding

Template Link: Team/namespace Onboarding

Tag and Business metadata¶

This standard change enables the creation, modification and deletion of Tags and Business Metadata on KaaS. Tags and Business metadata do not directly influence data records and configuration - it adds a layer, which can help discovery and add context information about the data products. It is important to distinguish between the actual "object" management and the association/linking to specific topics and schemas. Tags and business metadata data objects cannot be modified if they are actively associated to topics and schemas, and can only be handled by the central team with the Stream Governance Pipeline - Overview. LOB teams are responsible for associating objects within their 'Data products' and can do it self-service via the Data Product Pipeline Template - Overview.

Objects can be requested by Teams and need to be assessed by the Product Owner to determine if they should fall under 'generic' tags and business metadata with reuse across Novo Nordisk and ownership by the platform, or if they should be prefixed with the specific Team/namespace name to remove confusion.

Change Handling & Operational Practicalities¶

These instructions guide the handling of the Tag and Business Metadata update change; - Platform Operators should notify the Product Owner, that there are open requests related to Tags and Business metadata. - Platform Operators can create the PR on the Governance Pipeline referencing an instance of this std CHG - Product Owner should always be final approver - In case of team prefixed (owned) tags, the Data- and/or Team owner should approve any changes

Template Link: Tag and Business Metadata

Terraform and Terraform provider confluent¶

Terraform (vendor: Hashicorp) and Terraform provider confluent (vendor: Confluent Inc.) are open-source infrastructure-as-code tools used to define the cloud and on-premises resources in human-readable configuration files, and in turn used in automated deployments of KaaS Pipelines.

Refer CIL Wiki page to know about the pipelines that are using the terraform technology in KaaS project.

The vendors of Terraform and Terraform Confluent Provider frequently update their products, which triggers the KaaS to take up these modifications into account at the NN environment depending on the implications.

### Risk assessment Impact of these tool updates to NN setup are assessed to be 'minimum to low' as these updates does not introduce any major code level changes or pre-defined configuration of above specified pipelines that will disturb the intended purpose of these ADO pipelines rather, provide us the latest capabilities and overcome from the vulnerabilities through patch updates from the vendors.

### Deployment and verification process Updates to Terraform and Terraform confluent provider is handled through Standard Changes in ServiceNow. However, normal change creation should still be considered for handling this task if the criticality is involved to the level that requires URS/Solution Design/Design Specification updates, and these risks are thoroughly assessed by Technical SME with KaaS Platform Architect, if the impact assessment requires before creating the type of change.

Verification of these implementations is handled via unscripted testing manner. Post upgrade, technical SME should execute all the relevant/impacted pipeline runs across the environments and attach the Platform-Cil repo evidence to confirm the successful implementation. Any manual CIL update, say for ex., Confluent-resource-template etc., also to be considered and produced as the evidence of confirmation in Change closure notes.

Note: Any new pipeline that uses Terraform and terraform provider in future also need to be considered for verification process.

Template link: Terraform and terraform confluent provider update

API Key rotation¶

This standard change enables the rotation of API keys across the KaaS ecosystem. API keys are used as programmatic authentication and authorization credentials across KaaS, and needs to be frequently rotated. API keys are stored as 'Secret variables' in ADO Libaries or as AWS secrets for use in pipelines and consuming/producing applications. The rotation does not require code/configuration updates to repositories, so there is no easy method to trace when it is performed. Only a limited amount of privileged users can implement rotation. To keep an easier overview, this standard change should be used whenever rotating, to help generate an overview of when it was last done, and help ensure it is done in an organized and structured manner.

Change Handling and Operational Practicalities¶

These instructions guide the handling of the API key update change, which can be split into 3 types;

'deployment' service accounts used by the LOB team/namespace pipeline (based on template) and stream-gov pipeline. These are automatically generated when the access-mgmt pipeline is executed and stored as AWS secrets.
- LOB Team/namespace must be informed of a 'service window' prior to rotation, as there may be a short interruption of service (e.g. their pipelines will not be able to connect and deploy configuration - no impact on existing configuration!)
'client' service accounts used by LOB team/namespace applications to produce/consume data records into Confluent Cloud. These are created with the 'KaaS - Service Account Creation' request via ServiceNow. The access-mgmt pipeline generates the service accounts that will be used as owners of the API credentials, which Platform Operators create in Confluent Cloud UI (or via CLI for Schema registry). They are stored locally with LOB teams/namespaces.
'TF_runner' - service account used for the platform pipelines confluent-infra and access-mgmt. This is a highly privileged Confluent Cloud credential, that is stored in as a secret variable under 'kaas-prod' in KaaS ADO project (and PIM safe NN_UMG_KAAS_BUILD). Rotated via Confluent Cloud UI.

Detailed Operation Guide on how to do rotation can be found here API Key rotation

Template link: API Key rotation

Standard Change Template for Normal Change creation¶

Normal change ticket in ServiceNow will require some minimum must essential information for preliminary review before proceeding with its pre-approval from Change manager for implementation.

These details are captured in a standard change ticket, where the baseline information is captured with examples below.

Guidance to create a change ticket:¶

The Change Manager (IT Service Manager) shall create the normal change in ServiceNow and prefill with as much information as possible. The change manager will ensure to capture and insert the relevant, detailed information from codebase SME, Platform Architect or DevOps engineer. The ITQC and SM should review the description before processing. The change will be approved by Service Manager in Authorize state before deployment in production. In absence of Service Manager, it could be approved by PO/Platform Architect for technical implementation in production. The change will be closed with closure notes referencing PRs, Pipeline-run and other relevant information by the IT Change Manager.

Description¶

High-level recap of what is being implemented as a part of this Change. Mention the reference of the related User Story See Example below:

This change will implement a more granular access support by the access-management pipeline, allowing permissions on a per topic level, instead of on a hierarchical namespace level.
e.g. karen_clinical_* --new support --> karen_clinical_milestones

Furthermore, this change also introduces a minor pipeline update (not affecting end users), by creating additional role-assignments for confluent client service account roles, 
supporting fewer environment deployments without breaking terraform state

Justification¶

Mention the current set-up. Provide the business justification and benefits for implementing the new set-up.

This extension of the access-management pipeline is implemented to support access (service account) access to a single topic,
ignoring the hierarchical way of granting access that was previously established. 
It is an extension of the existing logic, and the single-topic access is only possible if you add a parameter. (backwards-compatible).

The justification is based on a request/requirement from the namespace MDM, 
where they need to have client consume a single topic, without getting access to other topics within their pathing.

Implementation Plan¶

Use below Template and add the pre-implementation, implementation and post-implementation steps required for the execution of the Change. Provide the details in a sequential manner. Describe the Verification Strategy, Output of Implementation and Approval/Release Criteria. See Example below:

All changes will be implemented by KaaS Core Team. The change implementation will be done according to the Manage IT Infrastructure SOP, Q216301.

Since there are no changes to the qualified state of KaaS Infrastructure or core functionality from this change, a separate Implementation Plan will not be drafted.
All the planning, verification and conclusion will be handled through ADO and ServiceNow.

Specify the pre-implementation steps required for the change.
Specify the implementation steps for the change 
Specify the post-implementation activities to be performed for the Change.


Verification strategy:
Platform-Changes wiki instruction is followed, implementation is peer-reviewed and behaviour is checked. ADO references to PRs and Pipeline runs will be captured in the Closure notes.

Output of Implementation: 
This change will allow client service accounts to only have permissions to a single topic (ignoring the hierarchical namespace logic) if parameter is added. This is a requirement from one of our business namespace teams in MDM.

Approval and Release criteria:

ITQC and SM will review the closure comments and related documents updated via Pull requests. 
This change will be closed in ServiceNow post verification of closure notes. 

QA will not be involved in the formal review and approval of CHG process in ServiceNow as there is no significant changes to existing functionality and no modification/alteration is done on any URS items related to this change.

Risk & Impact analysis¶

Assess the potential effects of the change on services and users. Identify potential risks, their probability and severity, and develop strategies to mitigate them. Mention the impact if this change implementation is unsuccessful. Mention the impact to existing ITRA.

Mention the details of the products impacted and the associated detailed product impact

Impact analysis:

Products impacted due to this change:
Access Management:                   No Impact
Confluent Cloud:                     No Impact
Team Provisioning Pipeline:          Impacted*
Data Product Pipeline Template:      No Impact 
Stream Governance Pipeline:          No Impact
Pilot Application:                   No Impact 
Azure Pipeline Agent:                No Impact
Logging:                             No impact

*Actual configuration on Confluent Cloud is not affected. This is only an update to the pipeline, deploying the configuration and/or provisioning resources.

Team Provisioning Pipeline: 
URS: N/A
Solution Design: N/A
DS: N/A
Operation Guide: Minor
Code: added logic for allowing optional true/false flag on a single, specific topic (backwards compatible)
CIL: updated

Risk Analysis: 

Confluent Cloud and it's configuration, including environments, clusters network, are not affected by this change. Data Products and their associated pipelines are not affected.

The change will extend the functionality of the current access-management pipeline.

This change follows std. guidelines, processes and component configuration.

No change to ITRA v4.0 due to this change.

Hence the risk of this implementation is identified to be 'Low'. 

Backout plan¶

Specify the steps that will be carried out in order to roll back to initial state. See Example below:

Revert the implementation done via Pull Requests and remove the boundary Condition from the DevOpsUserPolicy if required.

Test Plan¶

Insert references to formalized test plans from ADO for scripted testing or reference unscripted testing as shown in example below. Attach the testing evidence directly in the Change for the Unscripted Testing. See Example below:

An unscripted testing approach is used, with a deployment of the pilot app and then do UI verification of correct granularity/level of access and new config items.

A copy of the test has also been done on pre-prod. The Testing evidence are attached to the Change.