Connect alerts to a Slack channel

You can use a Slack channel to receive alert messages. A bot sends messages to the Slack channel, and all channel members can see the alert messages.

Application scenarios

Alert messages from OCP are pushed to a bot in the Slack channel, which then distributes the alert messages to the group. To connect to the Slack bot, you only need to provide the bot's Webhook URL. Additionally, you can use enhanced features, such as defining message Markdown format, boldening alert names, and displaying alert information in a list.

Prerequisites

• Ensure the current OCP user has the following permissions:

  • Any resource permission from Cluster Maintenance, Tenant Maintenance, OBProxy Maintenance, Arbitration Service Maintenance, Binlog Service Maintenance, or Host Maintenance.
  • All menu permissions in the Alert Center.

• You have created a Slack channel and added a Webhook to it.

• After obtaining the Webhook, ensure network connectivity. You can send a test message using the following command:

  # YOU_WEBHOOK_URL_HERE: The URL for the Webhook, for example, https://hooks.slack.com/services/XXX/XXX/XXX.
  curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' YOU_WEBHOOK_URL_HERE
  

Considerations

The length limit for Slack messages is typically about 3,000 characters. There is no explicit official frequency limit, but platform-level throttling may apply. It is recommended to enable aggregation configuration in the OCP alert channel to send multiple messages as one.

Procedure

1. In the left navigation bar, click Alert Center.

2. On the Alert Center page, click Alert Channel.

3. On the Alert Channel tab, click Create Channel.

4. Configure the Slack alert channel.

   1. Basic configuration.

      Option	Required	Description
      Channel Type	Yes	Select Slack from the drop-down list.
      Channel Name	Yes	Customize the channel name for easy identification later.
      Webhook URL	Yes	The access address for the Slack bot. For how to obtain it, see Create a Slack channel and get its Webhook URL.

   2. Message configuration.

      1. Select the alert message format. The system supports two formats: Text and JSON.

      2. Configure the alert message as needed.

         1. Configure the alert message template. The system will display messages according to the template you configured.
         2. In Duplicate Alert Configuration, set the number of sends and the time interval for the same alert to control whether the alert message is sent repeatedly and the frequency of sending the alert message. The default alert interval is 600 seconds, with a minimum value of 300 seconds. The configured value must be greater than or equal to the detection cycle in the alert rule.
         3. Choose whether to enable the Alert Message Aggregation feature. Enabling message aggregation prevents excessive alerts from being sent within a short period, which could overwhelm important messages. After enabling alert message aggregation, the system will send alert messages according to the template and interval you configured.

      3. Configure the recovery message as needed.

         1. Configure the alert message template. The system will display messages according to the template you configured.
         2. Choose whether to enable the Recovery Message Aggregation feature. Enabling message aggregation prevents excessive recovery messages from being sent within a short period, which could overwhelm important messages. After enabling recovery message aggregation, the system will send recovery messages according to the template and interval you configured.

5. Click Send Test Message to test whether the configured alert channel can successfully send alert messages.

6. If the test message is sent successfully, click Submit to complete the configuration.

FAQ

Message Throttling

How to handle message sending failures caused by throttling, such as: "Due to the large volume of activity, we will not display some messages from this incoming webhook integration.", or its English equivalent: "Due to high activity, we will not display some messages from this incoming webhook integration."

The length limit for Slack messages is typically about 3000 characters. There is no explicit official frequency limit, but platform-level throttling exists. After throttling, all messages sent to Slack will fail, posing a certain risk. Solutions:

• Enable the Aggregation feature for the alert channel: Similar alerts or the same alert are sent only once within the aggregation period.
• Push Throttling: OCP's push settings provide throttling strategies, supporting throttling by day or hour (throttling dimension is based on push configuration + alert channel).
• Console Adjustment: Modify the ocp.alarm.event.resend-interval-seconds system configuration. By default, an alert event is generated every 3 minutes.

The above steps can effectively reduce the number of alert messages and lower the likelihood of alerts being throttled; however, a sudden large number of different alerts may still be throttled.

Network Unreachable

How to handle the issue where the user environment cannot directly access the Slack API?

First, use curl to test whether a test message can be sent. If the network is unreachable, debug the connectivity between OCP and the webhook link.

Upgrade Migration

How to update alert pushes after an upgrade or migration?

Slack Webhook is a new method for alert pushing. In theory, no action is required for HTTP, script, DingTalk, or other channels. If you need to switch to the Slack channel, simply add it.

Response Verification Information Failure

How to resolve a Response verification information failure?

Setting up Response verification information failure is to prevent issues such as users not receiving alerts or alerts being missed due to non-standard script writing. It requires the alert script to output valid information exactly once (which can be an API response code or the API response body). OCP then verifies whether the information matches, supporting regular expression matching and JSON key-value pair matching.

The Slack bot API returns JSON content, so JSON key-value pair matching should be used. The Response verification information for the Slack group message channel should be set to: ok. Scripts should not output anything other than the Response (for example, the curl command should include the -s parameter to prevent progress information output).