Sample alert channel settings

This topic describes how to configure alert channels and alert notifications. Some details are easy to overlook but are frequently used in practice.

Application scenarios

The following scenarios involve the content of this topic:

1. Sending different alerts to different responsible persons.

2. Sending alert messages in JSON format.

3. Receiving only alert messages, not alert recovery messages.

4. Verifying whether an alert is sent successfully.

Alert channels and alert notifications

An alert channel is the medium through which alert messages are sent, such as to a DingTalk group or a customer's alert platform.

By configuring alert channels and alert notifications, you can send different alerts to different responsible persons (first sent via a specific channel and then mapped to the responsible person in another system, such as an overnight duty group or a DBA group).

Sending alert messages in a specified language

Currently, OCP supports three languages: Simplified Chinese, Traditional Chinese, and English. Alert messages support these three languages. To send messages in different languages, configure it during alert notification.

You can set different sending languages for the channel when subscribing to alert messages to meet the personalized message requirements of operations personnel. Currently supported settings are: zh_CN (Simplified Chinese), zh_TW (Traditional Chinese), and en_US (English).

Sending alert messages in JSON format

Messages are divided into two types: alert messages and recovery messages, and they are processed differently.

• Alert messages

  Currently, the OCP alert channel settings page allows you to configure alert message templates and aggregated message templates, which are applicable to DingTalk group messages. If you need to receive messages in JSON format, perform the following configuration.

  Note

  Since alert messages only need to be sent in one language, you can subscribe to alerts in that language when setting alert notification rules.

  1. Set both the alert message template and the aggregated message template to JSON format.

  2. Reference the ${message_json} message template variable when sending the message. ${message_json} is the JSON-formatted message ${message_json} set in the previous step. The message JSON has been escaped and cannot be re-wrapped; for example, {"content": ${message_json}} would not be a valid JSON string.

  Notice

  • If you did not set a JSON message template in Step 1, the message content to be set in Step 2 should be {"content": "${message_json}"}. Since ${message_json} is a string, not a JSON string, double quotes are required.
  • There may be unescaped characters in ${alarm_summary} and ${alarm_description}. For example, in OceanBase Database logs, alerts often contain quotation marks or line breaks. In such cases, the ${message_json} containing ${alarm_summary} and ${alarm_description} is not a valid JSON structure. Similarly, you must set the message content to {"content": "${message_json}"}.

• Restore messages

  Currently, you cannot configure the message restoration template via the console. You must modify the data in the MetaDB using the command line. The procedure is as follows:

  1. Log in to the ocp database of the ocp_meta tenant.

     obclient -hxxx.xxx.xxx.xxx -P2883 -ur***@ocp_meta#ob_cluster -p****** -Docp
     

  2. Update the recover_message_template_i18n field in the ocp2_alarm_channel table.

     Notice

     recover_message_template_i18n is of the string type and stores data in a JSON structure.

     ```sql update ocp2_alarm_channel set recover_message_template_i18n='{"recover_message_template_en_us":"[Alert Recovery Notification]\n- Target ${alarm_target}\n- Details ${alarm_description}\n- Recovered at ${alarm_resolved_at}","recover_message_template_zh_tw":"[Alert Recovery Notification]\n- Alert object ${alarm_target}\n- Alert details ${alarm_description}\n- Recovery time ${alarm_resolved_at}","recover_message_template_zh_cn":"[Alert Recovery Notification]\n- Alert object ${alarm_target}\n- Alert details ${alarm_description}\n- Recovery time ${alarm_resolved_at}"}' where id=; ```

  The default configuration is as follows:

  # English template Restore message template
  [Alarm Recover Notification]
  - Target ${alarm_target}
  - Details ${alarm_description}
  - Recovered at ${alarm_resolved_at}
  
  ## Restore message template for simplified Chinese
  [OCP Alert Recovery Notification]
  - Alert target ${alarm_target}
  - Alert details: ${alarm_description}
  - Recovery time: ${alarm_resolved_at}
  
  ## Restore message template for Traditional Chinese
  [OCP Warning Recovery Notification]
  - Alert Target ${alarm_target}
  - Alert details: ${alarm_description}
  - Recovery time: ${alarm_resolved_at}
  

When configuring alert push, you can specify whether to receive alert recovery messages.

Verify the alert sending result

When the OCP version is lower than V3.3.0, success is determined solely by whether the alert was successfully sent to the target, without verifying the return result of the alert message. If the response contains an error code or a field indicating failure, OCP should perform verification. Therefore, in OCP V3.3.0, the ocp2_alarm_channel object includes a response_validation field for verification, with the following rules:

• If the return result is in JSON format, response_validation must also be configured as a JSON string (default is "{}") and the two must match: the returned alert content must include all key-value pairs from response_validation. For example, if the response_validation field for a DingTalk group channel is set to {"errcode":0,"errmsg":"ok"}, an error code of 0 indicates a successful sending result.

• If the return result is not JSON, response_validation must match the return result.

  Note

  The return result for a script channel includes standard error and standard output. The system prioritizes matching the script's standard error with response_validation. If the script's standard error (stderr) is empty, then the script's standard output (stdout) is used for matching with response_validation. The script must follow these specifications:

  1. If the script fails to send the alert message, it outputs the response result to standard error.
  2. If the script does not output any information to standard error, it outputs the response result only to standard output. Printing other logs in the script may cause misjudgment as a sending failure.

After upgrading OCP, you need to:

1. Set a reasonable value for response_validation.

2. Check if the alert channel script outputs logs; modify it to output the response body only to standard error or standard output.

Configuring HTTP channels or script channels via the console requires setting the "Response Validation Information" field. Other types of alert channels have fixed built-in configurations and do not require console configuration.