Send alert messages to a DingTalk group

You can set a DingTalk chatbot to send alert messages to a DingTalk group and all group members can view the alert messages.

Scenarios

Alert messages of OceanBase Cloud Platform (OCP) are pushed to the DingTalk chatbot. Then, the chatbot sends the alert messages to the DingTalk group. To allow alert messages to be pushed to the DingTalk chatbot, you only need to provide the webhook URL of the DingTalk chatbot.

You can use the following enhanced features:

• Mention (@) a group member: You must configure the mobile number of the member to be mentioned. Members of the group can mute group messages and only the one that is mentioned will receive the notification.
• Add a signature: OCP adds a signature to alert messages based on the provided key to avoid malicious message sending. This is a more secure notification method.
• Define the Markdown format for messages: You can apply boldface for alert names and display alert messages in a list.
• Set a proxy: If network communication between the intranet and the Internet is not supported, you must set a network proxy. This way, alert messages are sent to the proxy first, and then the proxy forwards the alert messages to the DingTalk group.

Prerequisites

• Make sure that you have the following permissions:

  • Resource Permissions: Cluster Read-only, Tenant Read-only, OBProxy Read-only, Arbitration Service Read-only, Binlog Service Read-only, or Host Read-only permission
  • Menu Permissions: Permission on the All menu of the Alert Center

• You have created and configured a DingTalk chatbot. The API of DingTalk Standard Edition supports a cumulative number of 10,000 calls per month. Each chatbot can send at most 20 messages to a group per minute. If more than 20 messages are sent, messages will be throttled for 10 minutes.

  For information about how to connect to a DingTalk chatbot, see the DingTalk chatbot documentation.

• You have obtained the webhook URL of the chatbot. After you configure the DingTalk chatbot, you will obtain its webhook URL, which is required when you configure a DingTalk alert channel in OCP. To ensure the security of alert messages, keep this webhook URL confidential.

  After you obtain the webhook URL of the DingTalk chatbot, make sure that the network connection is normal. You can run the following command to send a test message for verification:

  curl 'https://oapi.dingtalk.com/robot/send?access_token=Webhook token of your chatbot' \
  -H 'Content-Type: application/json' \
  -d '{"msgtype": "text","text": {"content":"DingTalk chatbot test message"}}'
  

Considerations

The preceding example shows that the alert message is delivered by the API of DingTalk. The access token is the unique identifier of the DingTalk chatbot in the API and must be kept properly. If the access token is leaked, attackers can request the API to maliciously send messages such as advertisements and illegal content. The DingTalk chatbot provides three types of security setting methods and you must choose one.

• Custom keyword: Only alert messages containing the keyword are sent to the group. This method is not actually secure. If the webhook URL is leaked, the API of DingTalk may be requested to maliciously send alert messages.
• IP address (or CIDR block): Only alert messages from the specified IP address are sent. This method has O&M risks. If the region of OCP changes, you must modify the configuration of the DingTalk chatbot.
• Signature: A signature is added to alert messages based on the key. DingTalk will verify whether the signature is correct. This prevents malicious message sending. This method is recommended.

Procedure

1. In the left-side navigation pane, choose Alert Center.

2. On the alert details page, click the Alert Channel tab.

3. On the Alert Channel tab, click Create Channel.

4. Configure the DingTalk alert channel.

   1. Basic settings

      Parameter	Required?	Description
      Channel Type	Yes	Select DingTalk from the drop-down list.
      Channel Name	Yes	The name of the channel for identification.
      Webhook URL	Yes	The access URL of the DingTalk chatbot. For information about how to obtain the access URL, see the DingTalk chatbot documentation.
      Signature Key	No	The DingTalk chatbot provides three security setting methods. You can obtain the signature key after you choose the signature security setting method.
      @Specified User	No	The entered mobile number must be the one used by the member to log on to DingTalk. When an alert is generated, this member will be mentioned in the group. You can enter multiple mobile numbers separated with commas (,).
      Proxy	No	The network proxy. The format is http:ip:port in OceanBase Cloud of a version earlier than V4.2.0, and is http://ip:port in OceanBase Cloud V4.2.0 and later. Notice If the HTTP proxy does not support HTTPS, you must change https in the webhook URL of the DingTalk chatbot to http.

   2. Message settings

      1. The default message format is text in OceanBase Cloud of a version earlier than V4.2.1, and it is in Markdown format since OceanBase Cloud V4.2.1.

      2. You can modify the message structure. Here is a sample of the message structure of the default format:

         Set Alert Message Format to Markdown. You can enter ${} to introduce a variable.

         ### OCP alert notification ${alarm_name}
         - Level: ${alarm_level}
         - Alert object: ${alarm_target}
         - Summary: ${alarm_summary}
         - Generation time: ${alarm_active_at}
         - Description: ${alarm_description}
         - OCP link: ${alarm_url}
         

5. You can use other features.

   Alert message aggregate: You can enable the channel aggregate feature to avoid repeatedly sending the same alert within a period. Since OCP V4.1.0, an alert is generated every 3 minutes by default. After you enable channel aggregate, an alert is sent only once every 10 minutes (600 seconds).

6. Click Submit.

FAQ

Message throttling

What do I do if an alert message fails to be sent due to message throttling? The keyword is as follows: {"status":1111,"wait":5,"source":"x5","punish":"deny","uuid":"xxx","errcode": 130101,"errmsg": "send too fast, exceed 20 times per minute"}.

The API of DingTalk Standard Edition supports an accumulative number of 10,000 calls. Each chatbot can send at most 20 messages to a group per minute. If more than 20 messages are sent, messages will be throttled for 10 minutes. After message throttling is started, all messages destined for the DingTalk group will fail. This incurs risks. You can take the following actions to resolve the issue:

• Enable the alert channel aggregate feature. This way, similar and identical alerts are sent only once within the aggregate period.
• Configure push throttling. You can configure throttling by day or hour in the push settings of OCP. You must configure push settings and the alert channel.
• Change the value of the ocp.alarm.event.resend-interval-seconds system parameter in the CLI. By default, an alert event is generated every 3 minutes.

The preceding actions can effectively reduce the number of alert messages and the probability of alert messages being throttled. However, if a large number of different alerts are generated instantaneously (more than 20 alert messages are sent within 1 minute), alert messages will be throttled.

Network disconnection

What do I do if I cannot directly access the API of DingTalk?

Use curl to check whether a test message can be sent. If no, click Create Channel on the alert details page and configure a network proxy in the Proxy field in the Basic Settings section.

Upgrade and migration

What do I do to update the alert push settings after an upgrade or migration?

In OCP of a version earlier than V4.0.1, alert messages are pushed to DingTalk through the HTTP or script channel. In OCP V4.0.1 and later, we recommend that you add a built-in DingTalk channel. To avoid verification result sending failures, observe the following rules:

• If you use the HTTP or script channel and alert messages are sent to a DingTalk group, we recommend that you use the built-in DingTalk channel and configure the channel based on this topic.

• If alert messages are sent to the alert platform of the customer and then to the DingTalk group, you can use the original alert channel and do not need to change it to the built-in DingTalk channel.

Response verification failure

What do I do if response verification fails?

Alert messages may fail to be received or may not be sent when an invalid alert script is used. To avoid these issues, response verification will fail if the script is nonstandard. The alert script provides only one valid response, which can be a response code or response body of the API. OCP verifies whether the response matches the specified regular expression or JSON key-value pair. The response returned by the API of the DingTalk chatbot is in the JSON format. Therefore, you must specify a JSON key-value pair for verification. The response verification information of the DingTalk alert channel must be in the {"errcode":0,"errmsg":"ok"} format. The script channel should not output other content except the response. For example, you must add the -s parameter in the curl command to avoid generating progress information.