Skip to content

Business Metadata and Tags

Stream Governance Pipeline (Platform Level)

Below specifications are in the context of the "Stream Governance" feature of Confluent Kafka.

  1. A new pipeline must be created in the KaaS ADO project to manage the "tags and metadata objects" on the overall platform level.
  2. Only users with "KaaS Platform Administrator" or "KaaS Platform Operator" role should be able to run this pipeline.
  3. The pipeline must have individual stages for managing the "tags and metadata objects" per environment (ex. DEV, TST, QA, PRD).
  4. Exactly one manual approval must be configured against each of the pipeline stages that deploy the configurations to DEV, TST, QA and PRD environments.
  5. It should be possible for the pipeline to perform the following operations in relation to the management of the "tags and metadata objects":
    • Add a new "Tag" object.
    • Add a new "Business Metadata" object.
    • Edit an existing "Tag" object (name can not be updated).
    • Edit an existing "Business Metadata" object (name can not be updated).
    • Delete an existing "Tag" object.
    • Delete an existing "Business Metadata".
  6. Pipeline stages should be able to implement the changes in an atomic fashion. Either all configurations are applied or none are applied. This is relevant in scenarios such as deletion of objects when they are already in use.
  7. A separate folder should be created in the source code repository to manage the "tags and metadata objects". Following folder structure is to be maintained.
1
2
3
4
5
6
7
    /configurations
        /tags
            file-with-some-tags.json                (example)
            another-file-with-some-more-tags.json           (example)
        /business-metadata
            file-with-some-business-metadata.json           (example)
            another-file-with-some-more-business-metadata.json  (example)
  1. It should be possible to define the "tags and metadata objects" in multiple json files. The pipeline should scan the relevant folder for all files available and then consolidate all the configuration available prior to deploying those.

Applying Business Metadata and Tags (Application Level)

Below specifications are in the context of the "Stream Governance" feature of Confluent Kafka.

  1. Application pipeline template must be extended, so that a pre-configured "tag" on the platform can be added against the following Kafka objects:
    • Kafka Topic
    • Schema Object (on object/version level)
    • Schema Record (on overall record level)
    • Schema Field (on a field level)
  2. Application pipeline template must be extended, so that a pre-configured "business-metadata" on the platform can be added against the following Kafka objects:
    • Kafka Topic
    • Schema Object (on object/version level)
  3. A separate folder should be created in the source code repository to manage the configurations, including those of "tags and metadata". Following folder structure is to be maintained.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    /configurations
        /topics
            resource-topic-name.json
            example-employee-topic.json
            file-with-multiple-topics.json
        /schemas
            /avro
                resource-name-example.avsc
                avro-example-employee.avsc
            /protobuf
                protobuf-example-employee.proto
            /json
                json-example-employee.json
            /metadata
                resource-name.json
                example-employee.json
                file-with-multiple-schemas.json
  1. Tags and business metadata against "Kafka Topics" should be described in the files used to define the topic itself, i.e. at location /configurations/topics/*.json
  2. Tags and business metadata against "Schema Object (on object/version level)" should be described in the files at location /configurations/schemas/metadata/*.json
  3. Tags against "Schema Record (on overall record level)" and "Schema Field (on a field level)" should be embedded in the files used to define the schema itself, i.e. at location /configurations/schemas/avro/*.avsc for avro schemas (and similarly for protobuf and json).
  4. Exactly one manual approval must be configured against each of the pipeline stages that deploy the configurations to DEV, TST, QA and PRD environments. This would mean that for each deployment environments, the pipeline should have the following pipeline stages:
    • "terraform plan" stage : describing the changes to be implemented in the subsequent "terraform apply" stage, this includes all changes except the ones related to metadata which can not be defined at this step due to terraform provider's technical limitations.
    • "terraform apply" stage : where most of the changes are implemented. Only this stage should require a manual approval.
    • "terraform plan" stage : where remaining changes are implemented. This stage should not require a manual approval and should just run right after successful completion of the previous pipeline stage.