Description
This topic describes an alert that is generated when the client connection usage rate of OBProxy exceeds the threshold.
Principle
| Parameter | Value |
|---|---|
| Monitoring metric | client_connection_used_percent |
| Metric source | The number of client connections is collected from the metrics of OBProxy: odp_current_session in http://ip:2884/metrics (count). The maximum number of client connections is collected from the system parameters of OBProxy: SHOW proxyconfig LIKE |
| Collected metrics | odp_current_session, obproxy_client_max_connections |
| Monitoring expression | 100 * sum(odp_current_session{sessionType="client",@LABELS}) by (@GBLABELS) / sum(obproxy_client_max_connections{@LABELS}) by (@GBLABELS) |
| Collection interval | 30 seconds |
Rule information
| Monitoring expression | Meaning of the monitoring metric | Default threshold | Detection interval | Elimination interval |
|---|---|---|---|---|
| client_connection_used_percent > 85 | The ratio of the number of client connections to the maximum number of client connections | 85 | 15 seconds | 5 minutes |
Alert information
| Alert trigger method | Alert level | Scope |
|---|---|---|
| Based on the expression of the monitoring metric | Critical | Host |
Alert template
Alert summary
- Template: ${alarm_target} ${alarm_name}
- Example: alarm_template_id=0:obproxy_cluster=TEST-1:host=xxx.xxx.xxx.xxx OBProxy client connection usage exceeds threshold
Alert details
- Template: OBProxy cluster: ${obproxy_cluster}, host: ${host}, alert: ${alarm_name}. The client connection usage rate is ${value_shown}, which exceeds the threshold of ${alarm_threshold} %.
- Example: OBProxy cluster: TEST, host: xxx.xxx.xxx.xxx, alert: OBProxy client connection usage exceeds threshold. The client connection usage rate is 90%, which exceeds the threshold of 85%.
Alert recovery
- Template: Alert: ${alarm_name}, OBProxy client connection usage rate: ${value_shown}
- Example: Alert: OBProxy client connection usage exceeds threshold, OBProxy client connection usage rate: 80%
Impact on the system
- The response time for business requests increases.
- Business requests are affected after being piled up. For example, SQL requests may not respond for a long time or may be timed out.
Possible causes
- A large number of business requests.
- OBServer responds slowly and the number of connections is piled up.
Solution
Handle the alert by using the alert plan.
- Execute the plan to quickly restore the alert. For more information, see Alert plans.
- Retry the plan. Make sure the plan can continue to be executed to ensure that the original issue is resolved.
- Roll back the plan. Contact OceanBase Technical Support for assistance.
You can increase the maximum number of client connections and expand the memory of OBProxy by 20%. Modify the parameters:
client_max_connectionsandproxy_mem_limited. The specific commands are as follows:ALTER proxyconfig SET client_max_connections = xxx; ALTER proxyconfig SET proxy_mem_limited = xxx;