Alert-related concepts

This topic describes the concepts of alert targets, alert scope, alert rules, alerts, alert rule groups, alert levels, templates, alert aggregation, and alert elimination.

Alert target

An alert target is the subject being monitored by an alert task. It uniquely identifies the object associated with an alert. It can be an OceanBase cluster, a server, or a service.

The format of an alert target depends on the dimensions of the alert item and can be a label value or a combination of label values. For example, obregion=obocp:svr_ip=*.*.*.* identifies a server on an OceanBase or OCP cluster.

Alert scope

The alert scope defines the scope of an alert, which is consistent with the metric scope. For example, CPU usage exceeding limits can refer to different scopes, such as overall cluster usage exceeding limits, tenant CPU usage exceeding limits, or disk usage exceeding limits.

The alert scope includes the following values:

• OceanBase cluster (ObCluster)

• OceanBase tenant (ObTenant)

• Application cluster (AppCluster)

• Service

• Host

• Process (reserved)

Alert rule

An alert rule is the metadata of an alert, including the alert target, name, trigger rules, and alert details configuration.

Alert items can be classified into two types based on their generation method:

• Expression trigger: Alerts created on the console, where the alert rule engine generates the rules based on monitoring metrics.

• Custom trigger: Alerts triggered by other components.

For alerts triggered by expressions, the alert rule expressions are configured in the alert item. For alerts triggered by custom triggers, the alert rule expressions are empty.

Alert

An alert is the state of an alert rule occurring on an alert target.

For example, if an alert item named alarm_b is generated on server A, an alert signal is triggered every minute during the fault period. However, in OCP, this is considered a single alert. On the alert event page, there is only one record.

Alert item grouping

Each alert item can be assigned to multiple groups to facilitate the management of alert items and the distribution of alert messages.

Alert level

Each alert item has an associated alert level.

Templates

Templates are used to generate dynamic content based on variables at runtime. Templates can be used for the following purposes:

• Alert generation (configure the Alert Details Configuration template in an alert rule)

• Notification content (configure the Alert Message Template and Alert Message Aggregation Template)

• Channel parameters (such as the URL, header, and body content templates for HTTP channels)

Here are some template examples:

• alarm_summary: ${alarm_targe} CPU usage exceeds the limit

• alarm_description: ${alarm_targe} CPU usage has been continuously ${alarm_duration} above ${alarm_threshold}

For more information about the supported template variables, see OCP alert templates.

Alert aggregation

To avoid an alert storm caused by an excessive number of alerts, you can configure alert aggregation for an alert channel.

The aggregation rules are as follows:

• OceanBase log alerts are aggregated by alert type, log error code, and OceanBase cluster.

• Other OceanBase alerts are aggregated by alert type and OceanBase cluster.

• Application alerts are aggregated by alert type and alert target.

Alert elimination

The concept of alert elimination is equivalent to fault recovery. The perception of fault recovery may be triggered by the monitoring module notifying the alert service, or it may be automatically implemented by the alert service based on the alert elimination timeout.

The alert elimination logic is as follows:

• Each alert has a check cycle and an ignore cycle.

• The monitoring system detects that the alert meets the elimination conditions and calls the alert interface to mark the alert as eliminated.

• If the alert does not continue to be reported after the ignore cycle, it is considered that the alert has been eliminated.

Alert API

To facilitate the management of OceanBase resources and the development of your own applications, OCP provides HTTP APIs that conform to the REST specifications. Alert API describes the alert-related APIs, their syntax, and provides examples. The API documentation is targeted at developers. Before calling an API, make sure that you have thoroughly understood the concepts of the OCP platform.

Choose an appropriate method based on the connection method:

• API call description

• Alert API call example

Quick start

Alerts and alert notifications are important status monitoring features of OCP. Different roles take different actions when using the alert feature.

OCP provides general alert rules and templates. If you do not configure alert rules and templates, you can complete a simple alert configuration process by configuring alert channels and alert push.

The recommended alert configuration process is as follows:

1. Administrator operations:

   1. (Optional) Configure alert rules and templates. After deploying OCP, the administrator can manage existing alert rules and templates based on business needs.

   2. Configure alert channels. Set the alert push method (such as DingTalk, WeCom, and email), target, and message format.

   3. Configure alert push. Notify users of the alerts generated on the platform through the specified alert channel.

2. User operations:

   1. View alert events. OCP users can view alert events and notifications for different modules in daily use.

   2. (Optional) Mute alerts. OCP users can mute alert information.