ODP failure

ODP (obproxy) related failures mainly fall into two categories: one is the failure of the obproxy node itself, and the other is the failure of the load balancing components (usually F5, LVS, SLB, etc.) at the upper layer of a cluster composed of multiple obproxy nodes. This topic provides information on how to handle the failure of the obproxy node itself in emergency situations.

Emergency handling procedure

Failures caused by obproxy itself generally fall into the following situations:

• The obproxy process frequently exits due to the out-of-memory (OOM) error.

  If the obproxy process exits due to insufficient memory, you can find the following error message in the obproxy log: obproxy's memory is out of limit's 90%. You can modify the upper memory limit that the obproxy can occupy through the proxy_mem_limited parameter. If the obproxy memory usage exceeds this limit, the obproxy process exits. The value range of this parameter is [100 MB, 100 GB]. The modification immediately takes effect without the need for a restart.

  • If the obproxy is deployed through OCP, the default startup configuration already sets this parameter to 2 GB. In this scenario, the parameter value can be increased to 8 GB.

  • If the obproxy is deployed through the command line without explicitly specifying the memory size, the default value for proxy_mem_limited is 800 MB. In this scenario, you can try increasing the parameter value to 2 GB as follow:

    ALTER proxyconfig SET proxy_mem_limited = 2G;
    

• The obproxy cannot automatically restart due to the abnormal daemon process.

  In this scenario, you can manually restart the obproxy.