Send alert messages to a Feishu group

You can set a chatbot to send alert messages to a Feishu group and all group members can view the alert messages.

Scenarios

Alert messages of OceanBase Cloud Platform (OCP) are pushed to the Feishu chatbot. Then, the chatbot sends the alert messages to the Feishu group. To enable alert messages to be pushed to the Feishu chatbot, you only need to provide the webhook URL of the Feishu chatbot.

You can use the following enhanced features:

• Add a signature: OCP adds a signature to alert messages based on the provided key to avoid malicious message sending. This is a more secure notification method.
• Define the Markdown format for messages: You can apply boldface for alert names and display alert messages in a list.
• Set a proxy: If network communication between the intranet and the Internet is not supported, you must set a network proxy. This way, alert messages are sent to the proxy first, and then the proxy forwards the alert messages to the Feishu group.

Prerequisites

• You have created and configured a Feishu chatbot. A chatbot can receive at most 100 messages per minute and five messages per second. If this limit is exceeded, messages are throttled.

  For more information about how to connect to a Feishu chatbot, see the Feishu chatbot documentation.

• You have obtained the webhook URL of the chatbot. After you configure the Feishu chatbot, you will obtain its webhook URL, which is required when you configure a Feishu alert channel in OCP. To ensure the security of alert messages, keep this webhook URL confidential. After you obtain the webhook URL, run the following command to check whether the test message can be sent to the Feishu group:

  curl 'https://open.feishu.cn/open-apis/bot/v2/hook/webhook URL of your chatbot' \
  -H 'Content-Type: application/json' \
  -d '{"msg_type": "text","content": {"text":"Feishu chatbot test message"}}'
  

Considerations

The preceding example shows that the alert message is delivered by the API of Feishu. The token is the unique identifier of the Feishu chatbot in the API and must be kept properly. If the token is leaked, attackers can request the API to maliciously send messages such as advertisements and illegal content. The Feishu chatbot provides three security setting methods to ensure the security and reliability of alert messages.

• Custom keyword: Alert messages must contain the specified keyword.
• IP address allowlist: Only IP addresses (of OCP nodes) in the allowlist can send alert messages through the Webhook.
• Signature: A signature is added to alert messages based on the key. Feishu will verify whether the signature is correct. This prevents message tampering and malicious message sending. This method is recommended.

Procedure

1. In the left-side navigation pane, choose Monitoring and Alert > Alert.

2. On the alert details page, click the Alert Channel tab.

3. On the Alert Channel tab, click Create Channel.

4. Configure the Feishu alert channel.

   1. Basic settings

      Parameter	Required?	Description
      Channel Type	Yes	Select Feishu from the drop-down list.
      Channel Name	Yes	The name of the channel for identification.
      Webhook URL	Yes	The access URL of the Feishu chatbot. For information about how to obtain the access URL, see the Feishu chatbot documentation.
      Signature Key	No	The signature key. The Feishu chatbot provides three security setting methods. You can obtain the signature key after you choose the signature security setting method.
      @Specified User	No	The entered mobile number must be the one used by the member to log on to Feishu. When an alert is generated, this member will be mentioned in the group. You can enter multiple mobile numbers separated with commas (,).
      Proxy	No	The network proxy. The format is http:ip:port in OceanBase Cloud of a version earlier than V4.2.0, and is http://ip:port in OceanBase Cloud V4.2.0 and later. Notice If the HTTP proxy does not support HTTPS, you must change https in the webhook URL of the Feishu chatbot to http.

   2. Message settings

      1. The default message format is text in OceanBase Cloud of a version earlier than V4.2.1, and it is in Markdown format since OceanBase Cloud V4.2.1.

      2. You can modify the message structure. Here is a sample of the message structure of the default format:

         Set Alert Message Format to Markdown. You can enter ${} to introduce a variable.

         ### OCP alert notification ${alarm_name}
         - Level: ${alarm_level}
         - Alert object: ${alarm_target}
         - Summary: ${alarm_summary}
         - Generation time: ${alarm_active_at}
         - Description: ${alarm_description}
         - OCP link: ${alarm_url} 
         

5. You can use other features.

   Alert message aggregate: You can enable the channel aggregate feature to avoid repeatedly sending the same alert within a period. Since OCP V4.1.0, an alert is generated every 3 minutes by default. After you enable channel aggregate, an alert is sent only once every 10 minutes (600 seconds).

6. Click Submit.

FAQ

Message throttling

What do I do if an alert message fails to be sent due to message throttling? The keyword is as follows: {"code":9499,"msg":"too many request","data":{}}.

A Feishu chatbot can receive at most 100 messages per minute and five messages per second. If this limit is exceeded, messages are throttled. After message throttling is started, all messages destined for the Feishu group will fail. This incurs risks. You can take the following actions to resolve the issue:

• Enable the alert channel aggregate feature. This way, similar and identical alerts are sent only once within the aggregate period.
• Configure push throttling. You can configure throttling by day or hour in the push settings of OCP. You must configure push settings and the alert channel.
• Change the value of the ocp.alarm.event.resend-interval-seconds system parameter in the CLI. By default, an alert event is generated every 3 minutes.

The preceding actions can effectively reduce the number of alert messages and the probability of alert messages being throttled. However, if a large number of different alerts are generated instantaneously, alert messages will be throttled.

Network disconnection

What do I do if I cannot directly access the API of Feishu?

Use curl to check whether a test message can be sent. If no, click Create Channel on the alert details page and configure a network proxy in the Proxy field in the Basic Settings section.

Upgrade and migration

What do I do to update the alert push settings after an upgrade or migration?

In OCP of a version earlier than V4.0.1, alert messages are pushed to Feishu through the HTTP or script channel. In OCP V4.0.1 and later, we recommend that you add a built-in Feishu channel. To avoid verification result sending failures, observe the following rules:

• In OCP of a version earlier than V4.0.1, alert messages are sent to the Feishu chatbot through the script or HTTP channel and the signature security setting method is not supported. In OCP V4.0.1 and later, you can copy the webhook URL. For more information, see the description in other sections of this topic.

• If you use the HTTP or script channel and alert messages are sent to a Feishu group, we recommend that you use the built-in Feishu channel and configure the channel based on this topic.

Response verification failure

What do I do if response verification fails?

Alert messages may fail to be received or may not be sent when an invalid alert script is used. To avoid these issues, response verification will fail if the script is nonstandard. The alert script provides only one valid response, which can be a response code or response body of the API. OCP verifies whether the response matches the specified regular expression or JSON key-value pair. The response returned by the API of the Feishu chatbot is in the JSON format. Therefore, you must specify a JSON key-value pair for verification. The response verification information of the DingTalk alert channel must be in the {"StatusCode":0,"StatusMessage":"success"} format. The script channel should not output other content except the response. For example, you must add the -s parameter in the curl command to avoid generating progress information.