| Title | Kafka as a Service - Solution Design |
|---|---|
| Version | 6.0 |
| Approval Pull Request (PR) | PR229067 |
Solution Designs¶
Purpose & Scope¶
This solution design covers the entire KaaS platform and attempts to create an overview of the setup. The platform consists of 'Products' - technical components and deliverables, that together constitutes the entire platform service. Products are further split down into Solution designs, where requirements are traced to design. Technical design and other detailed documentation is stored with the source code in GIT, which is referenced from the Product Solution Designs.
Abbreviations and definitions¶
See Abbreviations and definitions
Overall Design¶
- Access Management spans across multiple products, where Azure Active Directory is used as identity provider. User onboarding is initiated from novoAccess
- Confluent Cloud is the managed Kafka Service delivered by Confluent. It consists of a subscription and control center, and an infrastructure-as-code 'Platform-Pipeline' which creates clusters, environment, network connectivity etc.
- 'Connectors' are pre-built, managed Apache Kafka connectors, that make it easier to connect to popular data sources and sinks (targets). The connectors are built by Confluent or their partners, and receive full support
- Teams Provisioning Pipeline onboards users and sets up resources and permissions for teams using the platform.
- Azure Pipeline Agent is a small, short-lived component that ensures connectivity from Azure DevOps pipelines into the private network of Novo Nordisk's Confluent Cloud. It is used by LOB teams/projects using the 'Data Product Pipeline Template', and in platform context the 'KaaS Pilot Application'.
- Data Product Pipeline Template (formerly known as 'Application Pipeline Template') gives 'teams' using the platform a boilerplate template to deploy their topics and schemas on the platform, accelerating and standardising the development process without relying on the central team.
- 'kaas-pipeline-templates', contains the terraform code and core logic
- 'kaas-assets-template', contains the pipeline stages and references all the logic in kaas-pipeline-templates when executed. This template is cloned and edited into team/namespace specific repos and pipelines.
- Pilot Application is used to simulate a 'team' using the platforms Kafka capabilities. It consists of two deliverables;
- KaaS MAP Pilot Application, which produces and consumes messages to topics and verifies schema validation.
- KaaS Pilot Data Product, which creates the topics and schemas used above and deploys the business metadata and tags
Note: KaaS Pilot Data Product is also be named as 'KaaS Pilot Application' in the above architecture diagram and same name is used across all the linkage documents and Git Repos respectively. Additional naming is done to avoid the misunderstanding as this not considered to be an application like that of KaaS MAP Pilot Application.
- Logging which consists of the MAP application 'KaaS Audit Log forwarder', which forwards audit logs - Configuration actions and changes on the platform, including topic retention changes - to a Novo Nordisk Splunk instance.
- Stream Governance Pipeline is used to create business metadata and tags that can be used by onboarding projects to link with topics, schemas and connectors.
Note! k6 is used for automating the integration test (IT) with the KaaS MAP Pilot Application producing/consuming messages, producing/consuming messages from the Confluent Cloud rest proxy using OIDC, and Performance testing the network during the failover testing ensuring high availability. The k6 scripts are stored in the GIT repo with the KaaS MAP Pilot Application.
Change Log¶
| Version no. | CHG no. | Approved Pull Request | Date | Change Description |
|---|---|---|---|---|
| 1.0 | CHG0100614 | PR111467 | See PR | New document |
| 2.0 | CHG0122187 | PR129688 | See PR | Stream Governance Pipeline details are updated |
| 3.0 | CHG0137496 | PR139926 | See PR | KaaS Pilot application is renamed as 'KaaS Pilot Data Product' |
| 4.0 | CHG0136055 | PR145250 | See PR | Updated 'Application Pipeline Template' to 'Data Product Pipeline Template' and associated link as part of ADO resource Refactor CHG |
| 5.0 | CHG0126415 | PR157981 | See PR | Removed association between Teams Pipeline and (Azure) Pipeline Agent from description |
| 6.0 | CHG0184608 | PR229067 | See PR | Updated with Connector details. Also added more accurate details to Data Product Pipeline resources. |