Sample alert channel settings

This topic describes how to configure alert channels and alert notifications. Some details are easy to overlook but are frequently used in practice.

Application scenarios

The following scenarios involve the content of this topic:

1. Sending different alerts to different responsible persons.

2. Sending alert messages in JSON format.

3. Receiving only alert messages, not alert recovery messages.

4. Verifying whether an alert is sent successfully.

Alert channels and alert notifications

An alert channel is the medium through which alert messages are sent, such as DingTalk group messages or the customer's alert platform.

By configuring alert channels and alert notifications, you can send different alerts to different responsible persons (first sent via a specific channel, then mapped to the responsible person in other systems, such as night duty groups or DBA groups).

Sending alert messages in a specified language

Currently, OCP supports three languages: Simplified Chinese, Traditional Chinese, and English. Alert messages support these three languages. To send messages in different languages, configure it during alert notification.

You can set different sending languages for the channel when subscribing to alert messages to meet the personalized message requirements of operations personnel. Currently supported settings are: zh_CN (Simplified Chinese), zh_TW (Traditional Chinese), and en_US (English).

Sending alert messages in JSON format

Messages are divided into two types: alert messages and recovery messages, and they are processed differently.

• Alert messages

  Currently, the OCP alert channel settings page allows you to configure alert message templates and aggregated message templates, which are applicable to DingTalk group messages. If you need to receive messages in JSON format, perform the following configuration.

  Note

  Since alert messages only need to be sent in one language, you can subscribe to alerts in that language when setting alert notification rules.

  1. Set both the alert message template and the aggregated message template to JSON format.

  2. Reference the ${message_json} message template variable when sending the message. ${message_json} is the JSON-formatted message ${message_json} set in the previous step. The message_json is escaped and cannot be wrapped again; for example, {"content": ${message_json}} would not be a valid JSON string.

  Notice

  • If no JSON message template is set in Step 1, the message content to be set in Step 2 should be {"content": "${message_json}"}. Since ${message_json} is a string, not a JSON string, double quotes are required.
  • There may be unescaped characters in ${alarm_summary} and ${alarm_description}. For example, in an OceanBase Database log alert, the alert content always contains quotation marks or line breaks. In this case, the ${message_json} containing ${alarm_summary} and ${alarm_description} is not a valid JSON structure. Similarly, you need to set the message content to {"content": "${message_json}"}.

• Restore messages

  The template for restoring messages is not currently configurable via the GUI. You must modify the data in the MetaDB using the command line. The procedure is as follows:

  1. Log in to the ocp database of the ocp_meta tenant.

     obclient -hxxx.xxx.xxx.xxx -P2883 -ur***@ocp_meta#ob_cluster -p****** -Docp
     

  2. Update the recover_message_template_i18n field in the ocp2_alarm_channel table.

     Notice

     recover_message_template_i18n is of the string type and stores data in a JSON structure.

     ```sql update ocp2_alarm_channel set recover_message_template_i18n='{"recover_message_template_en_us":"[Alert Recovery Notification]\n- Target ${alarm_target}\n- Details ${alarm_description}\n- Recovered at ${alarm_resolved_at}","recover_message_template_zh_tw":"[Alert Recovery Notification]\n- Alert object ${alarm_target}\n- Alert details ${alarm_description}\n- Recovery time ${alarm_resolved_at}","recover_message_template_zh_cn":"[Alert Recovery Notification]\n- Alert object ${alarm_target}\n- Alert details ${alarm_description}\n- Recovery time ${alarm_resolved_at}"}' where id=; ```

  The default configuration is as follows:

  # English template Restore message template
  [Alarm Recover Notification]
  - Target ${alarm_target}
  - Details ${alarm_description}
  - Recovered at ${alarm_resolved_at}
  
  ## Restore message template for simplified Chinese
  [OCP Alert Recovery Notification]
  - Alert target ${alarm_target}
  - Alert details: ${alarm_description}
  - Recovery time: ${alarm_resolved_at}
  
  ## Restore message template for Traditional Chinese
  [OCP Alert Recovery Notification]
  - Alert Target ${alarm_target}
  - Alert details: ${alarm_description}
  - Recovery time: ${alarm_resolved_at}
  

When configuring alert push, you can specify whether to receive alert recovery messages.

Verify the alert delivery result

When OCP is earlier than V3.3.0, success is determined solely by whether the alert is successfully sent to the target, without verifying the return result of the alert message. If the response contains an error code or a field indicating failure, OCP should perform verification. Therefore, starting from OCP V3.3.0, the ocp2_alarm_channel API adds a response_validation field for verification. The validation rules are as follows:

• If the return result is in JSON format, response_validation must also be configured as a JSON string (default is "{}") and the two must match: the returned alert content must include all key-value pairs from response_validation. For example, if the response_validation field for a DingTalk group channel is set to {"errcode":0,"errmsg":"ok"}, an error code of 0 indicates a successful delivery.

• If the return result is not JSON, response_validation must match the return result.

  Note

  The return result of a script channel includes standard error and standard output. The script's standard error is matched against response_validation first. If the script's standard error (stderr) is empty, then its standard output (stdout) is matched against response_validation. The script must follow these specifications:

  1. If the script fails to send the alert, it outputs the response result to standard error.
  2. If the script does not output any information to standard error, it outputs the response result only to standard output. Printing other logs in the script may cause misjudgment as a delivery failure.

After upgrading OCP, you need to:

1. Set a reasonable value for response_validation.

2. Check if the alert channel script outputs logs; modify it to output the response body only to standard error or standard output.

Configuring HTTP channels or script channels via the console requires setting the "Response Validation Information" field. Other types of alert channels have fixed built-in configurations and do not require console configuration.