Description
This alert is triggered when the usage of client connections of an OBProxy exceeds the threshold.
Principle
| Parameter | Value |
|---|---|
| Metric | client_connection_used_percent |
| Source | The number of current client connections is collected from the value of the OBProxy monitoring metric odp_current_session at http://ip:2884/metrics. The maximum number of client connections allowed is collected from the system parameters by using the SHOW proxyconfig LIKE statement. |
| Collected metric | odp_current_session, obproxy_client_max_connections |
| Metric expression | 100 * sum(odp_current_session{sessionType="client",@LABELS}) by (@GBLABELS) / sum(obproxy_client_max_connections{@LABELS}) by (@GBLABELS) |
| Collection cycle | 30 seconds |
Alert rule
| Metric expression | Metric description | Default threshold | Detection cycle | Time before clearance |
|---|---|---|---|---|
| client_connection_used_percent > 85 | The percentage of current client connections to the maximum number of client connections allowed. | 85 | 15 seconds | 5 minutes |
Alert information
| Trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the metric | Critical | Host |
Alert templates
Overview
- Template: ${alarm_target} ${alarm_name}
- Example: alarm_template_id=0:obproxy_cluster=TEST-1:host=xxx.xxx.xxx.xxx. The usage of client connections of the OBProxy exceeds the threshold.
Details
- Template: OBProxy cluster: ${obproxy_cluster}. Host: ${host}. Alert: ${alarm_name}. The usage of client connections reaches ${value_shown}, which exceeds the threshold ${alarm_threshold}%.
- Example: OBProxy cluster: TEST. Host: xxx.xxx.xxx.xxx. Alert: The usage of client connections of the OBProxy exceeds the threshold. The usage of client connections reaches 90%, which exceeds the threshold 85%.
Impact on the system
- The response time of business requests increases.
- The backlog of business requests results in exceptions, such as no responses to SQL queries and connection timeout.
Possible causes
- A large number of business requests are initiated.
- Prolonged connection occupation is caused due to slow response of the OBServer node.
Solutions
Execute the alert clearance plan.
- Execute the alert clearance plan to fix the alerted issue. For more information, see Execute the alert clearance plan.
- Retry the alert clearance plan. Make sure that the plan is executed to solve the original issue.
- Roll back the alert clearance plan. You can contact OceanBase Technical Support for assistance.
Increase the maximum number of connections allowed for an OBProxy and increase the memory for the OBProxy by 20% by modifying the
client_max_connectionsandproxy_mem_limitedparameters.The sample commands are as follows:
ALTER proxyconfig SET client_max_connections = xxx; ALTER proxyconfig SET proxy_mem_limited = xxx;