Push alert messages through HTTP

If your system has mature alert management features, you can push alert messages in OceanBase Cloud Platform (OCP) to an HTTP API.

Scenarios

Here are two common scenarios:

1. Alert chatbots provided by various cloud platforms, such as the DingTalk chatbot and Feishu chatbot, support sending custom messages and provide a webhook (API). Alert messages will be sent when an API request is received in the required message format.
2. The user side provides an alert API. Generally, no signature or special field (such as the current timestamp) needs to be added to the API. Alert messages will be sent after a message in the required format is pushed to the API.

Prerequisites

An API is provided for receiving alert messages. You can write a simple command to test the API. The request content varies based on the actual situation.

curl -XPOST 'API provided by the customer' \
 -H 'Content-Type:application/json; charset=utf-8' \
 --data '{"message":"Alert test message"}' \
 --compressed \
 --insecure

If the API uses authentication, such as Basic Auth, you can add the authentication information -H'Authorization: Basic xxxx' in the header.

Technical mechanism

Alert messages will be sent to the HTTP API upon an HTTP request. For DingTalk, the request fields are as follows:

• API address: the webhook URL of DingTalk.
• Request method: POST. Most APIs for receiving alert messages are POST APIs.
• Header: the header of an HTTP request, for example, Content-Type:application/json; charset=utf-8 or Authorization: Basic xxxx.
• Request body: the request parameters, including the API format, fields to be transmitted, meanings of the fields, and whether the fields are nullable. If a field is non-nullable and the corresponding field does not exist, you must specify the default field. For DingTalk, the request body is {"msgtype": "markdown","markdown": {"content":"${message}","title":"${alarm_name}"}}.
• Proxy: If the API is a public IP address and network communication between the intranet and the Internet is not supported, you must set a network proxy. Alert messages are sent to the proxy, and then the proxy forwards the alert messages to the API, such as the DingTalk API.

Considerations

1. Make sure that the API does not require complex request parameters, such as the special requirement to add a signature or the current timestamp.
2. The API must return necessary response content, for example, success (non-JSON message),{"err_code":0,"message":"success"}(JSON message). This way, OCP can easily determine whether alert messages are successfully sent. For more information, see the FAQ section of this topic.

Procedure

1. In the left-side navigation pane, choose Monitoring and Alert > Alert.

2. On the alert details page, click the Alert Channel tab.

3. On the Alert Channel tab, click Create Channel.

4. Configure the HTTP alert channel. For DingTalk, you must configure the following information:

   1. Basic settings

      Parameter	Required?	Description
      Channel Type	Yes	Select HTTP from the drop-down list.
      Channel Name	Yes	The name of the channel for identification.
      Recipient Field	Yes	Select User ID.

   2. Channel settings

      Parameter	Required?	Description
      Proxy	No	If the DingTalk API can be accessed over the network, you do not need to set a proxy.
      Request Method	Yes	POST
      URL Template	Yes	The webhook URL of DingTalk.
      Header Template	Yes	Content-Type:application/json; charset=utf-8. Separate different headers with line breaks.
      Body Template	Yes	{"msgtype": "text","text": {"content":"${message}"}}
      Response Verification Information	Yes	{"errcode":0,"errmsg":"ok"}

   3. Message settings

      1. The default message format is text in OceanBase Cloud of a version earlier than V4.2.1, and it is in Markdown format since OceanBase Cloud V4.2.1.

      2. You can modify the message structure. Here is a sample of the message structure of the default format:

         Set Alert Message Format to Markdown. You can enter ${} to introduce a variable.

         ### OCP alert notification ${alarm_name}
         - Level: ${alarm_level}
         - Alert object: ${alarm_target}
         - Summary: ${alarm_summary}
         - Generation time: ${alarm_active_at}
         - Description: ${alarm_description}
         - OCP link: ${alarm_url}
         

5. You can use other features.

   Alert message aggregate: You can enable the channel aggregate feature to avoid repeatedly sending the same alert within a period. Since OCP V4.1.0, an alert is generated every 3 minutes by default. After you enable channel aggregate, an alert is sent only once every 10 minutes (600 seconds).

6. Click Submit.

FAQ

Response verification failure

What do I do if response verification fails?

You must set the response verification information since OCP V3.3.0. Alert messages may fail to be received or may not be sent when an invalid alert script or API response (the 200 response code is returned even if an alert message fails to be sent) is used. To avoid these issues, response verification will fail if the alert script or API response is nonstandard. The alert script provides only one valid response, which can be a response code or response body of the API. OCP verifies whether the response matches the specified regular expression or JSON key-value pair. If the verification information does not match the response returned by the API, OCP determines that the alert message fails to be sent and tries to resend the alert message. However, if the user has received the alert message but the API considers that the message sending fails, the user may receive the alert message repeatedly. The verification rules are as follows:

• If the response is in the JSON format, the response verification information must be set to a JSON string, which is "{}" by default. If the response returned by the API contains all key-value pairs in the response verification information, it is considered that the two match. For example, if the API returns {"success flag 1":"success","error code":0,"current timestamp":"2023-09-12T10:00:00+8:00"}, the response verification information can be in any of the following formats: {"success flag 1":"success","error code":0}, {"success flag 1":"success"}, or {"error code":0}.

• If the response is not in the JSON format, the response verification information must be the same as the response. For example, if the response returned by the API is success, the response verification information must be set to success. If the response returned by the API is success:2023-09-12T10:00:00+8:00, the response verification information must be set to a regular expression success.*.

• If the response is enclosed with braces ({}), it is considered that the response is in the JSON format. In this case, if the response verification information is not a JSON string, a misjudgment is caused.

Response timeout of the alert API

The default timeout period for OCP to send an alert message is 10 seconds. If the response time of the alert API exceeds 10 seconds, OCP determines that the alert message fails to be sent and tries to resend the alert message. The parameters are described as follows:

• ocp.alarm.send.once-timeout-ms: the timeout period for sending an alert message. The default value is 10000 ms.
• ocp.alarm.send.once-retry-times: the number of retries for an alert message that fails to be sent. The default value is 1.