Examples of alert channel configuration

This topic describes how to configure an alert channel and alert push and provides some useful tips.

Application scenarios

This topic provides configuration examples in the following scenarios:

1. Send alerts to different users.

2. Send alert messages in the JSON format.

3. Receive only alert messages, but not alert clearance messages.

4. Verify whether an alert is sent.

Alert channel and alert push

An alert channel is a medium for sending alert messages, for example, sending messages to a DingTalk group or sending messages to the alerting platform of a customer.

You can configure an alert channel and alert push to send different alerts to different owners. Specifically, alerts are first sent through the specified channel and then assigned to corresponding owners such as the night duty group or DBA group in another system.

Send alert messages in a specified language

OceanBase Cloud Platform (OCP) supports alert messages in English, Simplified Chinese, and Traditional Chinese. If you need to send messages in a specified language, specify the language in the alert push settings.

You can specify different languages for different message channels to meet the personalized requirements of O&M personnel. The available language options include zh_CN (Simplified Chinese), zh_TW (Traditional Chinese), and en_US (English).

Send alert messages in the JSON format

Messages include alert messages and alert clearance messages. They are processed in different ways.

• Alert messages

  On the Alert Channel Configuration page of the OCP console, you can configure an alert message template and an aggregate message template. These templates apply to DingTalk group messages. To receive JSON alert messages, you must perform the following settings:

  Note

  If alert messages need to be sent only in one language, you can subscribe to alerts in a specific language when you configure alert push rules.

  1. Set the format of the alert message template and aggregate message template to JSON.

  2. Reference the message template variable ${message_json} in messages. The ${message_json} variable is the JSON message template ${message_json} configured in step 1. ${message_json} is already an escaped message string and cannot be packaged again. For example, {"content": ${message_json}} is an invalid JSON string.

  Notice

  • If the message templates are not set to the JSON format in step 1, the message content must be set to {"content": "${message_json}"} in step 2. The double quotations are inserted because ${message_json} is not a JSON string but rather a character string.
  • ${alarm_summary} and ${alarm_description} may contain unescaped characters. For example, OceanBase log alerts always contain quotation marks or line breaks. In this case, ${message_json} that contains ${alarm_summary} and ${alarm_description} is an invalid JSON format, and the message content also must be set to {"content": "${message_json}"}.

• Alert clearance messages

  The template for alert clearance messages cannot be configured in the OCP console. You need to modify the metadb data in the CLI. Perform the following steps:

  1. Log on to the ocp database of the ocp_meta tenant.

     obclient -hxxx.xxx.xxx.xxx -P2883 -ur***@ocp_meta#ob_cluster -p****** -Docp
     

  2. Update the recover_message_template_i18n field of the ocp2_alarm_channel table.

     Notice

     The recover_message_template_i18n field defines the templates of alert clearance messages in different languages in the JSON format. Make sure that the format is correct when you update the field.

     
     update ocp2_alarm_channel set recover_message_template_i18n='{"recover_message_template_en_us":"[Alarm Recover Notification]\n- Target ${alarm_target}\n- Details ${alarm_description}\n- Recovered at ${alarm_resolved_at}","recover_message_template_zh_tw":"【警示恢復通知】\n- 警示對象 ${alarm_target}\n- 警示詳情 ${alarm_description}\n- 恢復時間 ${alarm_resolved_at}","recover_message_template_zh_cn":"【告警恢复通知】\n- 告警对象 ${alarm_target}\n- 告警详情 ${alarm_description}\n- 恢复时间 ${alarm_resolved_at}"}' where id=;
     

  The following code shows the default configuration:

  # Alert clearance message template in English
  [Alarm Recover Notification]
  - Target ${alarm_target}
  - Details ${alarm_description}
  - Recovered at ${alarm_resolved_at}
  
  ## Alert clearance message template in Simplified Chinese
  【OCP 告警恢复通知】
  - 告警对象 ${alarm_target}
  - 告警详情 ${alarm_description}
  - 恢复时间 ${alarm_resolved_at}
  
  ## Alert clearance message template in Traditional Chinese
  【OCP 警示恢復通知】
  - 警示對象 ${alarm_target}
  - 警示詳情 ${alarm_description}
  - 恢復時間 ${alarm_resolved_at}
  

When you configure an alert push, you can configure whether to receive alert clearance messages.

Verify the alert sending result

If the OCP version is earlier than V3.3.0, the system only verifies whether an alert message is successfully sent to the destination, but does not verify the response of the alert message. However, verification is required if the response contains an error code or a failed field. Therefore, in OCP V3.3.0, the response_validation field is added to ocp2_alarm_channel for verification. The verification rules are as follows:

• If the response is in the JSON format, response_validation must also be set to a JSON string (“{}” by default). The response of the alert message must contain all the key-value pairs in response_validation. For example, for a DingTalk group message channel, if the response_validation field is set to {"errcode":0,"errmsg":"ok"}, the error code 0 indicates that the alert message is sent successfully.

• If the response is not a JSON string, it must be the same as the content of response_validation.

  Notice

  The response of a script channel contains a standard error and a standard output. The standard error in the script is preferentially matched against response_validation. If the standard error (stderr) of the script is empty, the standard output (stdout) of the script is matched against response_validation. The script conforms to the following rules:

  1. If the alert messages fail to be sent, the response is output to the standard error.
  2. If the program does not output any information to the standard error, only the response is output to the standard output. If the script outs some other logs, the system erroneously determines that a sending failure has occurred.

Therefore, after you upgrade OCP, you must perform the following operations:

1. Set response_validation to an appropriate value.

2. Check whether the script of the alert channel outputs logs. If yes, modify the script so that it only outputs the response body to the standard error or standard output.

To configure the HTTP channel or script channel on the GUI, you need to set the Response Verification Information parameter. Other types of alert channels have fixed built-in configurations and do not need to be configured on the GUI.