Prod and Preprod Organizations

Purpose

This page describes the purpose and rationale for having two separate Confluent Cloud accounts ("organizations"). The concept was formally introduced with the ADO Resource Refactor change, CHG0136055.

Background & Rationale

Kafka-as-a-Service ('KaaS') was originally qualified and implemented with a single organization and 4 different environment clusters - dev, tst, qa and prod. This is the "Novo Nordisk Prod" organization. While this gives separation of concern for LOB Teams using and developing on the platform, it was quickly identified that this could cause issues for the platform team maintaining and enhancing the platform, as some permissions and configuration are shared across the entire organization, and all environments could be viewed as 'production' as users were active in all environments, depending on the state and lifecycle of their data products.

In order to have a set of resources that could be used for exploring and testing platform features, e.g. access model changes, pipeline automation features, and new functionality like Flink (stream processing) without impacting existing users, a entirely separate account - 'organization' - was established to create a 'platform-only' setup. The new organization is called "Novo Nordisk PreProd".

Initial Scope

The Preprod organization account is not qualified and cannot be used for formal testing.

The initial purpose is to have an organization where new features released by Confluent can be 'opted-in' and verified in a controlled manner before planning a CHG to the prod organization. The The Preprod organization account is not qualified and cannot be used for formal testing. platform is also used for informal testing of pipeline functionality and behaviour, and will be used for evaluation of major features e.g. Kafka Connect and Flink.

It has been discussed that the preprod organization may be used for formal testing at a later point, if the right level of controls and a baseline is established.

Architecture

The Preprod organization account is setup very similar to the prod after the refactor change. This makes pipeline developers more confident in their code and configuration, and was also done in anticipation of the The Preprod organization account is not qualified and cannot be used for formal testing. setup being used for formal verification in the future.

The platforms are architected and deployed with a shared approach, and share some of the ADO resource code base, with the input configuration and pipelines being separate to ensure flexibility and control.

• modules repos hold the core implementation logic used across the organizations
• config repos are separated between prod and The Preprod organization account is not qualified and cannot be used for formal testing.
• here you define the selection, name and size of resources on the platform
• pipelines deploying the configuration are separated with different approval policies, to ensure strict control on production and more flexibility on The Preprod organization account is not qualified and cannot be used for formal testing.

• both setups capture all configuration and approvals at deployment, and store them in a separate 'CIL-repo'
• AWS-NN-KaaS-PRD holds terraform state files, secrets and components related to the Prod organization
• AWS-NN-KaaS-VAL holds terraform state files, secrets and components related to the Preprod organization