Error codes

This topic summarizes the errors that may occur during the use of OBD.

General errors

OBD-1000: Configuration conflict x.x.x.x: xxx port is used for x.x.x.x

Cause: Port conflicts occur in the configuration file.

Solution: Check and modify the configuration.

OBD-1001: x.x.x.x:xxx port is already used

Cause: The port has been occupied.

Solution: Check the configuration and change the port.

OBD-1002: Fail to init x.x.x.x path

Cause:

1. user in the configuration file (the current user by default, if unspecified) does not have the write permission on the corresponding directory.

2. home_path is not empty.

You can determine the cause based on the error information.

Solution:

For case 1, you can resolve the problem in two ways.

• Run the following command to add or modify user information:

  obd cluster edit-config <deploy name>
  

• Log on to the target server and grant the current account the write permission on the corresponding directory.

For case 2, you can also resolve the problem in two ways.

• Select another directory.

• If you are sure that the current directory can be cleared, you can use the -f option. OBD will clear this directory by using the current user.

OBD-1003: fail to clean x.x.x.x:xxx

Cause: user in the configuration file (the current user by default, if unspecified) does not have the write permission on the home_path directory.

Solution: You can resolve the problem in two ways.

• Run the following command to add or modify user information:

  obd cluster edit-config <deploy name>
  

• Log on to the target server and grant the current account the write permission on the corresponding directory.

OBD-1004: Configuration conflict x.x.x.x: xxx is used for x.x.x.x

Cause: Path conflicts occur in the configuration file.

Solution: Check and modify the configuration.

OBD-1005: Some of the servers in the cluster have been stopped

Cause: Some servers in the current configuration have been stopped, but subsequent operations require the services of all servers to be online.

Solution: Run the obd cluster start <deploy_name> --wop command to start all services without loading parameters.

OBD-1006: Failed to connect to xxx

Cause:

1. OceanBase Deployer (OBD) and the specified server are disconnected.

2. The corresponding component process has exited or does not provide service.

3. The account and password do not match.

Solution:

If the error is due to cause 1, you need to solve the network connection issue.

If the error is due to cause 2, you can try restarting the component first. If the startup still fails, please refer to the error of startup failure for troubleshooting, such as OBD-2002.

If the error is due to cause 3, it is likely that you have changed the password by executing SQL statements, and the account password is different from that stored in the configuration file. As a result, OBD cannot connect to the component. In this case, you can use any of the following two solutions:

1. Execute SQL statements to change the password back to that stored in the configuration file of OBD.

2. Run the vi ~/.obd/cluster/<deploy name>/config.yaml command to change the password to the one that is in use for the component.

OBD-1007: (x.x.x.x) xxx must not be less than xxx (Current value: xxx)

Cause: The configuration of the ulimit parameter does not meet the requirements.

Solution: You can modify the corresponding files in the /etc/security/limits.d/ directory and the limits.conf file in the /etc/security/ directory as needed.

OBD-1008: (x.x.x.x) failed to get fs.aio-max-nr and fs.aio-nr

Cause: OBD cannot obtain the asynchronous I/O (AIO) configuration from the server.

Solution: Check whether the current user has the privilege to view fs.aio-max-nr/fs.aio-nr.

cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr

OBD-1009: x.x.x.x xxx need config: xxx

Cause: A parameter is missing for the related service component.

Solution: Run the following command to open the configuration file. Then, add the missing parameter to the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

OBD-1010: x.x.x.x No such net interface: xxx

Cause:

1. The devname parameter cannot be obtained on the CLI.

2. The devname parameter cannot be obtained on the GUI.

Solution:

For the first case, run the following command to open the configuration file. Then, add or modify the devname parameter in the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

For the second case, choose Cluster Configuration > More Settings on the GUI. On the page that appears, set the devname parameter.

OBD-1011: (x.x.x.x) Insufficient AIO remaining (Avail: xxx, Need: xxx), The recommended value of fs.aio-max-nr is 1048576

Cause: The number of AIOs available in the system is less than that required by the database.

Solution: Run the following command to modify linux aio-max-nr:

sudo sysctl fs.aio-max-nr=1048576

OBD-1012: xxx

Cause:

1. A type conversion exception occurs. For example, a string is passed in for an int parameter.

2. The value of a parameter is beyond the allowed range. For example, the value range of rpc_port is 1025 to 65535. If the value configured for rpc_port is not within this range, an error is returned.

3. A parameter is missing. For example, a key parameter such as home_path is not configured.

Solution:

For the first case, check the parameter type and modify the value.

For the second case, check the passed-in parameter value and modify it.

For the third case, check the parameter configuration. If any parameter is missing, configure it.

OBD-1013: xxx@x.x.x.x connect failed: xxx

Cause (two most common causes):

1. The username or password was incorrect.

2. The connection timed out.

Solution:

For the first case, run the following command to open the configuration file. Then, modify the username or password in the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

For the second case, check the server configuration. For example, check whether the port is correct and whether the firewall is enabled.

If the problem persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OceanBase deployment errors

OBD-2000: x.x.x.x not enough memory

Cause: The memory is insufficient.

Solution: When OBD starts, the memory is strictly calculated based on MemAvailable. If any cached memory can be released, run the following command:

echo 3 > /proc/sys/vm/drop_caches

If the memory is still insufficient, run edit-config and then adjust memory_limt and system_memory. Ensure that the following condition is met: memory_limt/3 ≤ system_memory ≤ memory_limt/2.

Note

memory_limt cannot be lower than 8 GB. In other words, your available memory must be greater than 8 GB.

OBD-2001: server can not migrate in

Cause: The number of available units is smaller than --unit-num.

Solution: Modify the passed value of --unit-num. Run the following command to view the number of available units:

select count(*) num from oceanbase.__all_server where status = 'active' and start_service_time > 0

OBD-2002: failed to start x.x.x.x observer

Cause: There are multiple causes for this error. Two most common causes are as follows.

• memory_limit is lower than 8 GB.

• system_memory is too large or small. Generally, the following condition must be met: memory_limt/3 ≤ system_memory ≤ memory_limt/2.

Solution:

• If the problem is caused by either of the preceding reasons, take actions accordingly.

• If the problem persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-2003: not enough disk space for clog. Use redo_dir to set other disk for clog, or reduce the value of datafile_size

Cause: The disk usage exceeds the limit.

Solution: Adjust the storage of disks.

• For automatic deployment, the disk usage cannot exceed 72%.

• For manual deployment, the disk usage cannot exceed 64%, if the configuration is not modified.

Note

If redo_dir and data_dir are on the same disk, the space to be occupied by data files is included when the disk usage is calculated.

OBD-2004: Invalid: xxx is not a single server configuration item

Cause: The modified parameter is a global one and cannot be separately modified for a single server.

Solution: Place the parameter to modify under global.

OBD-2005: Failed to register cluster. xxx may have been registered in xxx

Cause: The cluster registration failed or the cluster has been registered.

Solution:

• Run the obd cluster edit-config command to open the configuration file. Then, set the value of the obconfig_url parameter to the URL of the correct config server.

• If the config server is correct and you want to forcibly override the registered cluster, add the -f option to the obd cluster start command.

• If the config server is correct, you can run the obd cluster edit-config command to open the configuration file. Then, modify the appname and cluster_id parameters to specify a new cluster name and ID for deployment.

OBD-2006: x.x.x.x has more than one network interface. Please set devname for x.x.x.x

Cause:

1. The devname parameter cannot be obtained on the CLI.

2. The devname parameter cannot be obtained on the GUI.

Solution:

For the first case, run the following command to open the configuration file. Then, add or modify the devname parameter in the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

For the second case, choose Cluster Configuration > More Settings on the GUI. On the page that appears, set the devname parameter.

OBD-2007: x.x.x.x xxx fail to ping x.x.x.x. Please check configuration devname

Cause: The nodes cannot ping each other.

Solution:

1. Check whether the NIC configuration is correct.

2. Check whether the network connection between the nodes is normal.

OBD-2008: Cluster clocks are out of sync

Cause: The clocks on the servers in the cluster are out of synchronization.

Solution: Synchronize the clocks on the servers.

OBD-2009: x.x.x.x: when production_mode is True, xxx can not be less then xxx

Cause: In the production mode, the values of parameters such as __min_full_resource_pool_mem and memory_limit cannot be smaller than specified values.

Solution:

• If you are deploying a non-production environment, run the following command to open the configuration file. Change the value of the production_mode parameter to False and run the restart command provided in the output.

  obd cluster edit-config <deploy_name>
  

• If you are deploying a production environment, run the following command to open the configuration file. Modify the __min_full_resource_pool_mem and memory_limit parameters to values greater than the specified values. Then, run the restart command provided in the output.

  obd cluster edit-config <deploy_name>
  

OBD-2010: x.x.x.x: system_memory too large. system_memory must be less than memory_limit/memory_limit_percentage

Cause: The value of the system_memory parameter is too large. Its value must be smaller than the result of memory_limit/memory_limit_percentage × total_memory.

Solution:

1. CLI: Run the following command to open the configuration file. Change the value of system_memory and run the restart command provided in the output.

   obd cluster edit-config <deploy_name>
   

2. GUI: Choose Cluster Configuration > More Settings and specify system_memory.

OBD-2011: x.x.x.x: fail to get memory info.\nPlease configure 'memory_limit' manually in configuration file

Cause: The server cannot obtain the memory information.

Solution:

1. CLI: Run the following command to open the configuration file. Specify memory_limit and run the restart command provided in the output.

   obd cluster edit-config <deploy_name>
   

2. GUI: Choose Cluster Configuration > More Settings and specify memory_limit.

Test-related errors

OBD-3000: parse cmd failed

Cause: The mysqltest initialization file is not an .sql file.

Solution: Check the --init-sql-files parameter.

OBD-3001: xxx.sql not found

Cause: The initialization file cannot be found during the initialization of mysqltest.

Solution: Check whether the file declared by --init-sql-files is located under the --init-sql-dir directory.

OBD-3002: Failed to load data

Cause: There are multiple causes for this error. Two most common causes are as follows.

1. The tenant has insufficient resources or is under excessive test stress.

2. An error occurred in the data build script.

Solution:

If the error is due to cause 1, you can use a tenant with larger resource specifications or adjust parameters such as warehouses and load-workers to reduce the test stress.

If the error is due to cause 2, you can rerun the test because the data build script is obtained from the TPC official website. If the issue persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-3003: Failed to run TPC-C benchmark

Cause:

1. The test process was stuck and then terminated due to timeout.

2. An error occurred for the TPC-C test command.

Solution:

• You can try to rerun the test directly, or you can adjust parameters such as terminals to reduce the test pressure before you rerun the test.

• If you did not use the obtpcc package provided on the OceanBase Database official website, use obtpcc for testing.

If the issue persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBAgent-related errors

OBD-4000: Fail to reload x.x.x.x

Cause: The http_basic_auth_password of the node is not the same as that stored in OBD, which causes OBD to fail to access OBAgent.

Solution: If the two passwords are the same, check whether an unsupported parameter is included among the modified options, or whether the name of a parameter is incorrect.

OBD-4001: Fail to send config file to x.x.x.x

Cause: (Check whether the error is caused by either of the reasons.)

• The disk space for the home_path directory on OBAgent is insufficient.

• user in the configuration file (the current user by default, if unspecified) does not have the write permission on the home_path directory on OBAgent.

Solution: You can resolve the problem in two ways.

• Run the following command to add or modify user information:

  obd cluster edit-config <deploy name>
  

• Log on to the target server and grant the current account the write permission on the corresponding directory.

ODP-related errors

OBD-4100: x.x.x.x need config "rs_list" or "obproxy_config_server_url"

Cause: The server cannot obtain the rs_list or obproxy_config_server_url information.

Solution: Run the following command to open the configuration file. Add or modify the rs_list or obproxy_config_server_url parameter and run the restart command provided in the output.

obd cluster edit-config <deploy name>

OBD-4101: failed to start x.x.x.x obproxy: xxx

Cause: Failed to start ODP.

Solution: Perform further analysis based on the error message.

Grafana-related errors

OBD-4200: x.x.x.x grafana admin password should not be 'admin'

Cause: The password of the admin user of Grafana cannot be admin.

Solution: Run the following command to open the configuration file. Add or change the password and run the restart command provided in the output.

obd cluster edit-config <deploy name>

OBD-4201: x.x.x.x grafana admin password length should not be less than 5

Cause: The password of the admin user of Grafana cannot be shorter than 5 characters in length.

Solution: Run the following command to open the configuration file. Add or change the password and run the restart command provided in the output.

obd cluster edit-config <deploy name>

OCP Express-related errors

OBD-4300: x.x.x.x: failed to query java version, you may not have java installed

Cause: OBD cannot obtain the Java information from the server.

Solution:

1. Install Java. For more information, see the How do I configure the Java environment before I deploy OCP Express? section in FAQ.

2. If Java has been installed, set the value of the java_bin parameter to the path of the executable file of Java.

OBD-4301: x.x.x.x: ocp-express need java with version xxx

Cause: The Java version on the server is earlier than needed.

Solution: Install Java of the version provided in the error message. If the target Java version has been installed, set the value of the java_bin parameter to the path of the executable file of Java.

OBD-4302: x.x.x.x not enough memory. (Free: xxx, Need: xxx)

Cause: The server has insufficient memory.

Solution:

• If the total memory available for the server is insufficient, run the obd cluster edit-config command to open the configuration file and modify the memory_limit parameter to a smaller value, or replace the server with one that has sufficient memory.

• If the remaining memory on the server is insufficient, run the following command to release cached objects that can be released.

  echo 3 > /proc/sys/vm/drop_caches
  

OBD-4303: x.x.x.x xxx not enough disk space. (Avail: xxx, Need: xxx)

Cause: The server has insufficient disk space.

Solution: Check and clean up the disk.

OBD-4304: OCP express xxx needs to use xxx with version xxx or above

Cause: OCP Express must be used with the corresponding component of the specified version.

Solution: Run the following command to open the configuration file. Modify the version of the corresponding component and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

OBD-4305: There is not enough xxx for ocp meta tenant

Cause: The log disk space and memory are insufficient for creating a meta tenant for OCP Express.

Solution:

• If the OceanBase cluster is deployed in Maximum Utilization mode on the GUI, or deployed by using the obd cluster autodeploy command on the CLI, we recommend that you clean up the disk and memory and then try again.

• If a resource specification is configured for the cluster, increase the values of the resource-related parameters for the oceanbase-ce component, for example, the memory-related parameters such as memory_limit and memory_limit_percentage and the log disk-related parameters such as log_disk_size and log_disk_percentage.

SQL-related errors

OBD-5000: sql execute failed

Cause: The SQL execution failed.

Solution: Determine the solution based on the actual situation.