Kafka as a Service - Verification Strategy¶

Purpose¶

This document describes the general Verification strategy for the Kafka as a Service (KaaS) Platform. Information about the initial release can be found in Verification Strategy - Platform Release 1.0 and in the Agile IT Implementation Plan and Report ref [1].

The verification will be done according to the Manage IT Infrastructure [2] and Verification of IT infrastructure [3]

Abbreviations and definitions¶

See Abbreviations and definitions

Verification Strategy considerations¶

In accordance with the initial Release & Implementation plan & Report, following aspects have been considered while developing this strategy:

The identified life cycle controls from the IT risk assessment. Refer to the latest released version of the ITRA for the application under the IT Risk Assessment Tab Kafka as a Service (KaaS) | Business Application | Novo Nordisk ServiceNow
Requirements created in accordance with ‘Manage IT Infrastructure’

Requirements related to Access Management is one of the key areas of interest, ensuring the correct roles and privileges assigned to teams/namespaces being onboarded and building assets on the platform - Pipeline Automation and its use in defining and and deploying configuration with a high level of control and transparency.

IT risk assessment¶

The IT risk assessment outcome can be summarised into two main areas that must be addressed and re-assessed when changed via implementation and verification activities:

Data considerations
- Data is the responsibility of the teams / namespaces using the platform Below mentioned agreement is in place to ensure roles and responsibilities are in place to support this setup.
  Refer KaaS - Project on-boarding agreement - Template - Confluent cloud is an iPaaS solution and assists in data transitioning. The platform as such does not store or hold the responsibility of storage of data. The onboarding teams/namespaces using the platform are responsible for DI and GDPR controls - The risk is mitigated by the F-00920036 KaaS - Project on-boarding agreement - Template v1.0 (section 4.1.1) that must be signed by projects before being onboarded - Two requirements have been created to support logging requirements (Platform and Audit Logs). More details here Logging - Overview - Security / privilege controls - RBAC setup/permissions and governance must be in place, to ensure separation of duties between platform activities, environments and teams maintaining their data streams and applications - Multiple Access Management requirements concerning novoAccess, AzureAD, SSO and Password management (PIM) have been created to mitigate this risk.

IT risk has been considered while developing the Requirements for the platform.

Requirements¶

Requirements have been created and each requirement has been assessed for test verification activities. These assessments are described as acceptance criteria in the requirements. The acceptance criteria act as the test scoping / coverage and therefore reduce the need for formal test strategies.

RS requirements are implicitly verified by the test done by Unit, Integration and OV testing. Test Traceability can be seen here - Test Traceability - Overview

KaaS Requirements can be found here.

The platform is made out of different products. The requirements are grouped into these products. For each product a solution design exists.

A product as defined in the RS will be a combination of different components. These components can consist of APIs, configurations, roles, environments, etc. The components are grouped into an ADO page per product and named Solution Design and holds Solution Design Requirements.

Products / Solution Designs¶

A list of Solution Designs can be found in the Solution Designs section. In the list a link can be found to each Solution Design. This includes a visualization of how they relate.

Pilot Application(s)¶

A Pilot application is created to simulate a customer team/namespace being onboarded to the platform, and using it as intended e.g. defining and deploying configuration to the platform.

The Pilot application is used to verify that accesses and connectivity is setup correct.

Verification Approach¶

This document will focus on providing the reader an understanding of: - Testing Approach Process Diagram - Verification approach across the platform - Environments - Test organisation & roles and responsibilities - How to understand traceability from Requirements to test

Process Diagram for Testing Approach¶

Link to Miro Board : Testing Approach - Miro

Verification approach across the platform¶

This is the verification approach for Products, with Significant and High impact changes to the already established functionality and purpose of the platform.

Test plans will be made per product
There can be multiple Test plans per product
- Some Products must have parts of the Product verified as a pre-requisite for other Products. E.g. IV has to be performed as it is a dependency for another Product, but OV can wait as OV has a dependency on yet another Product
Test Plans can include multiple Products
- At the time of creation of the Verification Strategy no such plans exists but this document allows for the flexibility to do so, if needed
Some Products are tested as part of other Products
- E.g. Azure Pipeline Agent is verified indirectly via Product testing of Team Provisioning Pipeline and KaaS Pilot Application
- Rationales for leveraging other Test to Verify a Product must be stated in Test Plans
A combined Test plan for Pilot Application(s) act as the end-to-end test as discussed in the Agile IT Implementation Plan and Report [1]
The verification approach along with the test types identified to take place for a given product must be stated in the Test Plans
- Section 'Overall Acceptance Criteria' below must be adhered to in Test plans
  - Rationales for deferring from the Overall 'Acceptance Criteria' must be stated in Test Plans
- Products must at least have Installation Verification in QA and Prod environments
- Products that have API code developed must have Code review, Unit Test and Integration Test
- Products that have Terraform code developed must have Code review of pipelines
Test plans can reference this Verification Strategy document to avoid duplication of information and sections
Test plans will list traceability for the Product requirements to test cases, as further described in section 'How to understand traceability from Requirements to test' below
Test plans will have a sequential order to be run in. This is defined in 'Table 1 Products, Test plans and Sequence of Test' as a number in column 'Test Plan ID + Sequence Number in ( )'

Release management¶

All Products must be verified before the Kafka as a Service Platform is released for use and available for customers / projects to be onboarded to the platform.

Releases are managed with ServiceNow.

Environments¶

Prepod organization

Informal testing is performed on prepod organization.

Prod Organization

Environment name	Purpose of env.	Who	Activities	Strategy (activities)
Development	Development and/or configuration and setup of system functionality and/or API’s	Project team API developers Confluent	Development	Development and/or configuration and setup according to the guidelines
Test	Test environment for system functionality and/or developed API’s	Project team API Developers	Test	Test of development and/or functionality according to guidelines. Verification documented in ADO via Product Test plans
QA	Quality assurance environment similar to the production environment	Project team System management	Formal testing	Full test and verification documented in ADO via Product Test plans
Production	Production environment, which is under configuration and change control.	System management	Operation Deployment of new API’s and system functionality	Operation and Maintenance

Table 2 - Environments

Test organisation & roles and responsibilities¶

The following roles are identified to support verification activities.

Roles	Responsibilities
PO	Review and approve Test plans
Platform Architect/Development Team (Tester)	Create Functional Test Cases Review and approve Test cases Execute Test Cases Create Test Defects
QC	Creates Test Plan Review and approve Functional Test runs Review and approve Test defects Create and perform: Test execution review,conclusion and approval Test case
IT QA	Supervision only - do not have any part in the verification activities

Table 3 - Test organisation & roles and responsibilities

Roles and responsibilities are described in the Wiki page KaaS Artefacts, Reviews and Approvals and Platform Team Roles & Responsibilities

Training of Testers¶

Training for using DevOps testing module has been done as peer training from Test Manager to testers.

In addition testers must be trained according to Kafka as a Service Azure DevOps Testing and Kafka as a Service Test Defect Management.

Training must be documented as per instructions on the WIKI page Kafka as a Service Azure DevOps Testing.

Agile IT Implementation Plan and Report [1] section 7.14 states that testers must be trained in Q0300381 - Validation Deviations. This is replaced by the local instruction Kafka as a Service Test Defect Management..

How to understand traceability from Requirements to test¶

Traceability is established: Test Traceability - Overview

between RS and Solution Designs requirements.
between specifications requirements to relevant verification test cases and test results.

Requirements and specifications are traceable via relations in Azure DevOps in the Solution Designs section of the Wiki.

Test case traceability is created by tracing test cases and test runs to requirements in a table overview to support Test Plan traceability. Traceability will be documented, attached to Test plan, and reviewed as part of the Test plans (before test execution) and in the 'Test execution review, conclusion and approval Test case' (after test execution) to list successful runs.

Test execution¶

Pre-requisites¶

Before testing is started the following pre-requisites must be in place:- - Test case is approved - Test plan is approved - Test plan must show traceability from RS -> SD -> Test case - RS, SD, FS, DS, Operation Guide must be in approved state via a Pull Request - Code review is approved via Pull Request, in accordance with the Code review guideline - CIL management is done in accordance with CIL - Overview

Defect Management¶

Defect management follows the standard Kafka Test Defect Management process defined in the Kafka WIKI: Defect Management

Overall Acceptance Criteria¶

Product testing must be completed in an environment before the Product testing can be initiated in the next environment.

Deployment of builds can be done through Dev, Test, QA and Prod environments at the same time, as long as it is verified that the build being released to Prod is verified in QA environment first and released for Prod.

All development is subject to code review, unit test and integration test
Informal testing is done in the prepod environment wherever possible and Formal Testing is done in the prod environment in all applicable clusters.
For Unscripted Testing the Test Scenarios are described in the Test Plan field of the Change and the related test evidence are attached to the Change.
For Scripted Testing the IV/pIV, OV Test Cases are reported in ADO

A test phase (from one environment to the next) is completed when:

All test cases have been executed and the results have been evaluated against relevant Acceptance Criteria and/or Expected Results
All Test Defects have been resolved and approved by relevant roles
The results have been reported in the 'Test execution review, conclusion and approval Test case' and acknowledged by relevant roles

References¶

F-00883817 KaaS Implementation Plan
Q216303 Manage IT Infrastructure
Q0779370 Verification of IT Infrastructure
Q187655 Manage Information Security in IT Solutions
Q204010 Manage data integrity in IT solutions
Q187218 Ownership of IT systems and IT infrastructure
F-00925673 KaaS-Design Review Report