Overview¶
Link to the Dashboard on Grafana: KaaS Confluent Metrics Grafana
Dashboard Configurations: KaaS-Confluent-Metrics-Dashboard-Panel
Visualizations¶
Confluent Server Status¶
Purpose: The purpose of this visualization is to retrieve the connection information for a specified Kafka broker ($kafka_id). The results would typically be useful for understanding the connection status, load, and overall health of the broker in terms of client interactions. Monitoring connection metrics is essential for ensuring optimal performance and for diagnosing issues related to connectivity.
Connector Status¶
Purpose: The purpose of this visualization is to provide information regarding the operational state of that connector, indicating whether it is functioning correctly or if there are any issues that need to be addressed. Monitoring connector status is vital for managing data integration tasks and ensuring that data flows smoothly between Kafka and external systems.
Confluent Kafka Server Successful Authentication Count¶
Purpose : The purpose of this visualization is to capture the delta count of successful authentications. Each sample is the number of successful authentications since the previous data point. The count is sampled every 60 seconds.
Confluent Kafka Server Partition Count¶
Purpose : The purpose of this visualization is to capture the partition count for a specific Kafka broker ($kafka_id). The result indicates how many partitions are assigned to the broker, which is critical for assessing the broker's role in the Kafka cluster and understanding the distribution of data across your Kafka infrastructure.
Confluent Kafka Server Consumer Lag Offsets¶
Purpose : The purpose of this visualization is to capture the total consumer lag offsets for each consumer group in a specific Kafka broker ($kafka_id). The results provide insights into how well consumer groups are keeping up with the data being produced to Kafka, which can help in monitoring the health of consumers and identifying any potential lag that needs to be addressed to ensure timely message processing.
Confluent Kafka Cluster Load Percent¶
Purpose : The purpose of this visualization is to capture the load percentage for a specific Kafka broker ($kafka_id). The resulting value provides insights into how much of the broker's capacity is being used, which can help in monitoring overall performance, diagnosing potential issues, and planning for capacity scaling if needed. The value is between 0.0 and 1.0.
Confluent Kafka Server Active Connection Count¶
Purpose : The purpose of this visualization is to capture the count of active authenticated connections.
Confluent Kafka Schema Registry Schema Operations Count¶
Purpose : The purpose of this visualization is to capture the delta count of schema related operations (create, delete and read). Each sample is the number of requests received since the previous data point. The count is sampled every 60 seconds.
Confluent Kafka Schema Registry Schema Count¶
Purpose : The purpose of this visualization is to capture the number of registered schemas.
Confluent Kafka Schema Registry Request Count¶
Purpose : The purpose of this visualization is to capture the delta count of requests received by the schema registry server. Each sample is the number of requests received since the previous data point. The count is sampled every 60 seconds
Confluent Kafka Server Request Count¶
Purpose: The purpose of this visualization is to capture the delta count of requests received over the network. Each sample is the number of requests received since the previous data point. The count is sampled every 60 seconds.
Write/Read Records Percentage¶
Purpose: The overall purpose of this visualization is to determine how well the Kafka instance is handling message flow by comparing the average number of received records per unit of time to the average number of sent records per unit of time. This analysis can be crucial for identifying performance issues or understanding processing bottlenecks.
Read of Topics (rate)¶
Purpose: The ultimate purpose of this visualization is to determine the total average size (in bytes) of messages sent by the specified Kafka server over the defined time window. This information can be vital for understanding the bandwidth usage and performance characteristics of your Kafka messaging system.
Write On Topics (rate)¶
Purpose: The purpose of this visualization is to display the average rate of bytes received by Kafka brokers over a defined rate interval for a specific Kafka ID and across all topics, aggregating the results across different brokers.
Request Count (rate)¶
Purpose: The purpose of this visualization is to show the average number of requests handled by a specific Kafka broker over a defined time period, aggregating this average across all principals interacting with the broker. This is useful for monitoring the load on a broker and understanding its request handling performance.
Dead Letter Queue Records Count¶
Purpose: The purpose of this visualization is to show the count of records stored in the dead letter queues of various Kafka Connect connectors across all jobs. It allows users to monitor how many records are failing and being sent to the DLQ, providing vital information for troubleshooting and processing within the Kafka ecosystem.
Connector Sent Records¶
Purpose: The purpose of this visualization is to show the count of records successfully sent by all Kafka Connect connectors across various jobs. It provides insights into the volume of data being processed and sent out, which is essential for monitoring the health and performance of the Kafka Connect infrastructure.
Partition Count Change (delta)¶
Purpose: The purpose of this visualization is to calculate the total absolute change in the number of Kafka partitions for a specific broker over the last 5 minutes. The result is useful for monitoring the stability of partition counts, which can provide insights into broker performance and topic management activities.
Topic Retained Bytes¶
Purpose: The purpose of this visualization is to provide a summary of the total number of bytes retained in all topics for a specified Kafka broker ($kafka_id). The results show how much data is being stored per topic, which is useful for monitoring storage usage and capacity planning in Kafka.
Topic Received Bytes¶
Purpose: The purpose of this visualization is to display the total number of bytes received for each topic on a specified Kafka broker ($kafka_id). The results can provide important insights into the activity and load on different topics, helping to monitor the incoming data flow for efficient resource management.
Topic Sent Bytes¶
Purpose: The purpose of this visualization is to display the total number of bytes sent to each Kafka topic for a specified Kafka broker ($kafka_id). The results give insights into how much data is being produced to different topics, which is important for monitoring the throughput and load on the broker, as well as for resource allocation and performance tuning.
Topic Received Records¶
Purpose: The purpose of this visualization is to display the total number of records received for each topic on a specified Kafka broker ($kafka_id). The results are useful for monitoring the data ingestion rates to different topics and understanding how much data is being processed by the broker from producers.
Kafka Server Sent Records¶
Purpose: The purpose of this visualization is to show the total number of records sent to each topic for a specified Kafka broker ($kafka_id). It provides insights into the data output rates for various topics and helps monitor the performance and load on the broker, enabling better management of resources and understanding of data flow within the Kafka ecosystem.
Topic Request Rate¶
Purpose: The purpose of this visualization is to retrieve the total number of requests received by a specified Kafka broker ($kafka_id) while summing across all user identities (principals) that are making requests to the broker. This information is essential for monitoring the overall load and demand on the Kafka broker, helping with performance analysis and resource management.