Skip to content

General Information

Pointers for Troubleshooting Cluster Load Percent and Consumer Lag Offset Alerts

  • Cluster Load Percent alert will get triggered if the average load for a cluster has increased above the defined limit.
  • Discuss with the service manager, after that decision would be taken for addition of any broker.
  • Consumer Lag Offset Alert will get triggered if the average Consumer Lag offset for a cluster has increased above the defined limit.
  • Check on Confluent cloud for the Consumer Lag details. Reach out to the owner of the topic for which lag is present.

KaaS - Cluster Load Percent PROD

KaaS - Cluster Load Percent VAL

KaaS - Cluster Load Percent TST

KaaS - Cluster Load Percent DEV

KaaS - Consumer Lag PROD

KaaS - Consumer Lag VAL

KaaS - Consumer Lag TST

KaaS - Consumer Lag DEV

Search Query

KaaS - Confluent Metrics Alert.txt

Alert Configurations

Owner : KaaSOpsAdmin

Settings for Cluster Load Percent - Alert Type : Scheduled - Schedule : Run on Cron Schedule - Cron Expression : */30 * * * * (every 30 minutes) - Time Range : Last 15 minutes - Expires : 30 days

Trigger Conditions - Trigger alert when : Number of Results - is greater than '0.7' - Trigger : Once - Throttle : Yes - Suppress triggering for : 60 minutes

Trigger Actions - Add to Triggered Alerts - Severity : High - Send email - To : KAASOPS@novonordisk.com - Priority : High - Subject : Splunk Alert: $name$ - Message : The alert condition for '$name$' was triggered. KaaS Wiki - Platform Alerts : https://dev.azure.com/novonordiskit/KaaS%20(Kafka%20as%20a%20Service)/_wiki/wikis/KaaS%20WIKI/161183/Platform-Alerts - Include : - Link to Alert - Link to Results - Trigger Condition - Allow Empty Attachments - Type : HTML & Plain Text

Settings for Consumer Lag Offset - Alert Type : Scheduled - Schedule : Run on Cron Schedule - Cron Expression : */30 * * * * (every 30 minutes) - Time Range : Last 15 minutes - Expires : 30 days

Trigger Conditions - Trigger alert when : Number of Results - is greater than '100' - Trigger : Once - Throttle : Yes - Suppress triggering for : 60 minutes

Trigger Actions - Add to Triggered Alerts - Severity : High - Send email - To : KAASOPS@novonordisk.com - Priority : High - Subject : Splunk Alert: $name$ - Message : The alert condition for '$name$' was triggered. KaaS Wiki - Platform Alerts : https://dev.azure.com/novonordiskit/KaaS%20(Kafka%20as%20a%20Service)/_wiki/wikis/KaaS%20WIKI/161183/Platform-Alerts - Include : - Link to Alert - Link to Results - Trigger Condition - Allow Empty Attachments - Type : HTML & Plain Text