| Title | Kafka as a Service Solution Design - Flink |
|---|---|
| Version | 1.0 |
| Approval Pull Request (PR) | PR241492 |
Solution Design - Flink¶
Purpose & Scope¶
This Solution Design covers Apache Flink support in Confluent Cloud — a cloud-native, powerful stream processing service that enables real-time data transformation, filtering, aggregation, and enrichment directly on Kafka topics.
Flink on Confluent Cloud integrates natively with Kafka, providing autoscaling, automatic version upgrades, security, and governance — without requiring teams to manage Flink Compute Pools or infrastructure.
While the platform team cannot guarantee correctness of user-developed Flink jobs, resource allocation is set up in a standardized way to isolate workloads between teams/namespaces.
In later phases, Flink deployments will be fully integrated into the Data Product Pipeline Template, enabling traceable, peer-reviewed, and automated deployments.
Abbreviations and definitions¶
See Abbreviations and definitions
Solution Design Requirements¶
| ID | Name |
|---|---|
| 1 | Flink Access |
| 2 | Flink Deployment |
| 3 | Flink Capabilities |
Flink Access¶
- Permissions for Flink SQL and API usage will be added via RBAC to existing team roles (e.g.,
FlinkDeveloper). - Developers can submit and manage Flink statements in Dev/Tst/QA via the Confluent Cloud UI
- Operators will manage Flink jobs in PRD.
Flink Deployment¶
Phase 1 (Initial Release)
- Flink SQL statements can be run via the Confluent Cloud UI.
Phase 2 (CI/CD Integration)
- Flink job definitions will be integrated into the Data Product Pipeline Template.
- This will enable:
- Version-controlled SQL and configuration in Git
- Peer review and approval gates in Azure DevOps
- Automated deployments via CI/CD pipelines
Flink Capabilities¶
- Stream & Batch Processing
Supports both real-time streaming and bounded snapshot batch queries. Batch jobs automatically release resources on completion.
- SQL Tables & Schema Handling
Kafka topics are automatically mapped to queryable Flink tables with schema metadata.
- Schema & Statement Evolution
- Statements are immutable after creation (properties and SQL).
- Compute pool and principal can be changed with a stop/resume cycle.
- Dependencies (tables, functions) are snapshotted at statement creation.
- Monitoring & Metrics
- Confluent Cloud Console supports job monitoring (status, compute pool, principal).
Related Documents¶
- Access Management
- Team Provisioning Pipeline
- Data Product Pipeline Template
- Confluent Flink Documentation
Traceability between Requirements¶
Traceability between Requirements (URS) and (SDR)¶
| ID | Work Item Type | URS (Status Dynamically Updated) | GIT SDR |
|---|---|---|---|
| 3185126 | URS | #3185126 | Solution Design Requirement: Flink |
Change Log¶
| Version no. | CHG no. | Approved Pull Request | Date | Change Description |
|---|---|---|---|---|
| 1.0 | CHG0201019 | PR241492 | Date of acknowledgement | New document |