Error codes

This topic summarizes the errors that may occur during the use of OBD.

General errors

OBD-1000: Configuration conflict x.x.x.x: xxx port is used for x.x.x.x

Cause: Port conflicts occur in the configuration file.

Solution: You can run the obd cluster edit-config command to open the configuration file, and view and modify the port configuration.

OBD-1001: x.x.x.x:xxx port is already used

Cause: The port has been occupied.

Solution: You can check the configuration and change the port by using one of the following methods:

• Method 1: If you use a configuration file for deployment, run the obd cluster edit-config command to modify the port configuration in the configuration file. Then, run the obd cluster start command to continue to start the cluster.

  Note

  For more information about the commands used in Method 1, see Cluster commands.

• Method 2: If you run the obd demo command for deployment, run the following command to specify the port. In this example, the mysql_port parameter of the oceanbase-ce component is specified:

  obd demo --oceanbase-ce.mysql_port=3881
  

  Note

  For more information about the commands used in Method 2, see Quick deployment command.

• Method 3: If you perform deployment on the GUI of OBD, you can change the port number on the Cluster Configuration page.

OBD-1002: Fail to init x.x.x.x path

Cause:

1. user in the configuration file (the current user by default, if unspecified) does not have the write permission on the corresponding directory.

2. home_path is not empty.

You can determine the cause based on the error information.

Solution:

For case 1, you can resolve the problem in two ways.

• Run the following command to add or modify user information:

  obd cluster edit-config <deploy name>
  

• Log on to the target server and grant the current account the write permission on the corresponding directory.

For case 2, you can also resolve the problem in two ways.

• Select another directory.

• If you are sure that the current directory can be cleared, you can use the -f option. OBD will clear this directory by using the current user.

OBD-1003: fail to clean x.x.x.x:xxx

Cause: user in the configuration file (the current user by default, if unspecified) does not have the write permission on the home_path directory.

Solution: You can resolve the problem in two ways.

• Run the following command to add or modify user information:

  obd cluster edit-config <deploy name>
  

• Log on to the target server and grant the current account the write permission on the corresponding directory.

OBD-1004: Configuration conflict x.x.x.x: xxx is used for x.x.x.x

Cause: Path conflicts occur in the configuration file.

Solution: Check and modify the configuration.

OBD-1005: Some of the servers in the cluster have been stopped

Cause: Some servers in the current configuration have been stopped, but subsequent operations require the services of all servers to be online.

Solution: Run the obd cluster start <deploy_name> --wop command to start all services without loading parameters.

OBD-1006: Failed to connect to xxx

Cause:

1. OceanBase Deployer (OBD) and the specified server are disconnected.

2. The corresponding component process has exited or does not provide service.

3. The account and password do not match.

Solution:

If the error is due to cause 1, you need to solve the network connection issue.

If the error is due to cause 2, you can try restarting the component first. If the startup still fails, please refer to the error of startup failure for troubleshooting, such as OBD-2002.

If the error is due to cause 3, it is likely that you have changed the password by executing SQL statements, and the account password is different from that stored in the configuration file. As a result, OBD cannot connect to the component. In this case, you can use any of the following two solutions:

1. Execute SQL statements to change the password back to that stored in the configuration file of OBD.

2. Run the vi ~/.obd/cluster/<deploy name>/config.yaml command to change the password to the one that is in use for the component.

OBD-1007: (x.x.x.x) xxx must not be less than xxx (Current value: xxx)

Cause: The configuration of the ulimit parameter does not meet the requirements.

Solution: You can modify the corresponding files in the /etc/security/limits.d/ directory and the limits.conf file in the /etc/security/ directory as needed.

OBD-1008: (x.x.x.x) failed to get fs.aio-max-nr and fs.aio-nr

Cause: OBD cannot obtain the asynchronous I/O (AIO) configuration from the server.

Solution: Check whether the current user has the privilege to view fs.aio-max-nr/fs.aio-nr.

cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr

OBD-1009: x.x.x.x xxx need config: xxx

Cause: A parameter is missing for the related service component.

Solution: Run the following command to open the configuration file. Then, add the missing parameter to the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

OBD-1010: x.x.x.x No such net interface: xxx

Cause:

1. The devname parameter cannot be obtained on the CLI.

2. The devname parameter cannot be obtained on the GUI.

Solution:

For the first case, run the following command to open the configuration file. Then, add or modify the devname parameter in the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

For the second case, choose Cluster Configuration > More Settings on the GUI. On the page that appears, set the devname parameter.

OBD-1011: (x.x.x.x) Insufficient AIO remaining (Avail: xxx, Need: xxx), The recommended value of fs.aio-max-nr is 1048576

Cause: The number of AIOs available in the system is less than that required by the database.

Solution: Run the following command to modify linux aio-max-nr:

sudo sysctl fs.aio-max-nr=1048576

OBD-1012: xxx

Cause:

1. A type conversion exception occurs. For example, a string is passed in for an int parameter.

2. The value of a parameter is beyond the allowed range. For example, the value range of rpc_port is 1025 to 65535. If the value configured for rpc_port is not within this range, an error is returned.

3. A parameter is missing. For example, a key parameter such as home_path is not configured.

Solution:

For the first case, check the parameter type and modify the value.

For the second case, check the passed-in parameter value and modify it.

For the third case, check the parameter configuration. If any parameter is missing, configure it.

OBD-1013: xxx@x.x.x.x connect failed: xxx

Cause (two most common causes):

1. The username or password was incorrect.

2. The connection timed out.

Solution:

For the first case, run the following command to open the configuration file. Then, modify the username or password in the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

For the second case, check the server configuration. For example, check whether the port is correct and whether the firewall is enabled.

If the problem persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-1015: Unable to confirm the primary-standby relationship, rerun with "--ignore-standby" option if you want to proceed despite the risks

Cause: The cluster or tenant involved in the current operation was engaged in a primary/standby relationship. However, an exception occurred in verifying the relationship when a command was being executed.

Solution: Check whether the current cluster or tenant has a standby tenant in another cluster. The check method varies based on whether the cluster is available.

• If the current cluster is available, you can run the following command to check whether a standby tenant exists. Here a cluster named test is used as an example.

  obd cluster tenant show test -g
  

• If the current tenant is unavailable, you can run the following command to view the primary/standby relationships of the current cluster or tenant. Here a cluster named test is used as an example.

  cat ~/.obd/cluster/test/inner_config.yaml
  

  Based on the returned result, run the following command on the corresponding cluster to check whether the primary/standby relationships still exist. Here a cluster named test-standby is used as an example.

  obd cluster tenant show test-standby -g
  

Take the following actions based on the check result. For more information about the commands involved in the following actions, see Cluster commands.

• If the current operation is an upgrade operation and the current cluster or tenant has a standby tenant, you can upgrade the cluster of the standby tenant first and then that of the primary tenant. This is because the cluster version of the standby tenant must not be earlier than that of the primary tenant. If the current cluster contains both primary and standby tenants, you can first run the obd cluster tenant switchover command to switch the primary tenant in the cluster to a standby tenant. After the clusters where the primary and standby tenants reside are upgraded, you can perform a switchover to switch the standby tenant back to the primary tenant.

  Note

  If the current cluster is undergoing an intra-version upgrade, you can run the command again with the check skipped by appending the --ignore-standby option to the command.

• If the current operation is not an upgrade operation, such as destroy, redeploy, or drop, and the current cluster or tenant has a standby tenant, you can remove the primary/standby relationship in one of the following ways:

  • Run the obd cluster tenant decouple command to decouple the standby tenant from the primary tenant. After the decoupling, the standby tenant becomes an independent regular tenant.

  • Run the obd cluster stop command to stop the cluster where the primary tenant resides and then run the obd cluster tenant failover command to perform a failover for the standby tenant. After the failover, the standby tenant becomes the new primary tenant.

  • Run the obd cluster tenant drop command to drop the standby tenant.

• If the current cluster or tenant has no standby tenant or if you can accept the risks caused by the unavailability of the standby tenant, you can run the command again with the check skipped by appending the --ignore-standby option to the command.

OceanBase deployment errors

OBD-2000: x.x.x.x not enough memory

Cause: The memory is insufficient.

Solution: When OBD starts, the memory is strictly calculated based on MemAvailable. If any cached memory can be released, run the following command:

echo 3 > /proc/sys/vm/drop_caches

If the memory is still insufficient, run edit-config and then adjust memory_limt and system_memory. Ensure that the following condition is met: memory_limt/3 ≤ system_memory ≤ memory_limt/2.

Note

memory_limt cannot be lower than 8 GB. In other words, your available memory must be greater than 8 GB.

OBD-2001: server can not migrate in

Cause: The number of available units is smaller than --unit-num.

Solution: Modify the passed value of --unit-num. Run the following command to view the number of available units:

select count(*) num from oceanbase.__all_server where status = 'active' and start_service_time > 0

OBD-2002: failed to start x.x.x.x observer

Cause: There are multiple causes for this error. Two most common causes are as follows.

• memory_limit is lower than 8 GB.

• system_memory is too large or small. Generally, the following condition must be met: memory_limt/3 ≤ system_memory ≤ memory_limt/2.

Solution:

• If the problem is caused by either of the preceding reasons, take actions accordingly.

• If the problem persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-2003: not enough disk space for clog. Use redo_dir to set other disk for clog, or reduce the value of datafile_size

Cause: The disk usage exceeds the limit.

Solution: Adjust the storage of disks.

• For automatic deployment, the disk usage cannot exceed 72%.

• For manual deployment, the disk usage cannot exceed 64%, if the configuration is not modified.

Note

If redo_dir and data_dir are on the same disk, the space to be occupied by data files is included when the disk usage is calculated.

OBD-2004: Invalid: xxx is not a single server configuration item

Cause: The modified parameter is a global one and cannot be separately modified for a single server.

Solution: Place the parameter to modify under global.

OBD-2005: Failed to register cluster. xxx may have been registered in xxx

Cause: The cluster registration failed or the cluster has been registered.

Solution: Check whether the appname parameter is configured for the cluster. If not, the cluster cannot be registered with Config Server. Take the following actions based on whether the cluster has been deployed.

• Case 1: If you want to register a to-be-deployed OceanBase cluster with Config Server, comment out the obconfig_url parameter, start the cluster, and then run the obd cluster edit-config command to configure the obconfig_url parameter. At present, a to-be-deployed cluster cannot be registered with Config Server.

• Case 2: If you want to register a started cluster with Config Server, check whether the obconfig_url parameter is correctly configured.

  • If obconfig_url is incorrectly configured, run the obd cluster edit-config command to open the configuration file and specify the correct Config URL for the obconfig_url parameter.

  • If obconfig_url is correctly configured and you want to forcibly overwrite the registered cluster, you can append the -f option to the obd cluster start command.

OBD-2006: x.x.x.x has more than one network interface. Please set devname for x.x.x.x

Cause:

1. The devname parameter cannot be obtained on the CLI.

2. The devname parameter cannot be obtained on the GUI.

Solution:

For the first case, run the following command to open the configuration file. Then, add or modify the devname parameter in the configuration file and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

For the second case, choose Cluster Configuration > More Settings on the GUI. On the page that appears, set the devname parameter.

OBD-2007: x.x.x.x xxx fail to ping x.x.x.x. Please check configuration devname

Cause: The nodes cannot ping each other.

Solution:

1. Run the obd cluster edit-config command to check whether the value of the devname parameter in the configuration file matches the actual network interface card (NIC) setting. You can use the ip addr command to check the mapping between IP addresses and NICs.

2. Check whether the network connection between the nodes is normal.

OBD-2008: Cluster clocks are out of sync

Cause: The clocks on the servers in the cluster are out of synchronization.

Solution: Synchronize the clocks on the servers.

OBD-2009: x.x.x.x: when production_mode is True, xxx can not be less then xxx

Cause: In the production mode, the values of parameters such as __min_full_resource_pool_mem and memory_limit cannot be smaller than specified values.

Solution:

• If you are deploying a non-production environment, run the following command to open the configuration file. Change the value of the production_mode parameter to False and run the restart command provided in the output.

  obd cluster edit-config <deploy_name>
  

• If you are deploying a production environment, run the following command to open the configuration file. Modify the __min_full_resource_pool_mem and memory_limit parameters to values greater than the specified values. Then, run the restart command provided in the output.

  obd cluster edit-config <deploy_name>
  

OBD-2010: x.x.x.x: system_memory too large. system_memory must be less than memory_limit/memory_limit_percentage

Cause: The value of the system_memory parameter is too large. Its value must be smaller than the result of memory_limit/memory_limit_percentage × total_memory.

Solution:

1. CLI: Run the following command to open the configuration file. Change the value of system_memory and run the restart command provided in the output.

   obd cluster edit-config <deploy_name>
   

2. GUI: Choose Cluster Configuration > More Settings and specify system_memory.

OBD-2011: x.x.x.x: fail to get memory info.\nPlease configure 'memory_limit' manually in configuration file

Cause: The server cannot obtain the memory information.

Solution:

1. CLI: Run the following command to open the configuration file. Specify memory_limit and run the restart command provided in the output.

   obd cluster edit-config <deploy_name>
   

2. GUI: Choose Cluster Configuration > More Settings and specify memory_limit.

Test-related errors

OBD-3000: parse cmd failed

Cause: The mysqltest initialization file is not an .sql file.

Solution: Check the --init-sql-files parameter.

OBD-3001: xxx.sql not found

Cause: The initialization file cannot be found during the initialization of mysqltest.

Solution: Check whether the file declared by --init-sql-files is located under the --init-sql-dir directory.

OBD-3002: Failed to load data

Cause: There are multiple causes for this error. Two most common causes are as follows.

1. The tenant has insufficient resources or is under excessive test stress.

2. An error occurred in the data build script.

Solution:

If the error is due to cause 1, you can use a tenant with larger resource specifications or adjust parameters such as warehouses and load-workers to reduce the test stress.

If the error is due to cause 2, you can rerun the test because the data build script is obtained from the TPC official website. If the issue persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-3003: Failed to run TPC-C benchmark

Cause:

1. The test process was stuck and then terminated due to timeout.

2. An error occurred for the TPC-C test command.

Solution:

• You can try to rerun the test directly, or you can adjust parameters such as terminals to reduce the test pressure before you rerun the test.

• If you did not use the obtpcc package provided on the OceanBase Database official website, use obtpcc for testing.

If the issue persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBAgent-related errors

OBD-4000: Fail to reload x.x.x.x

Cause: The http_basic_auth_password of the node is not the same as that stored in OBD, which causes OBD to fail to access OBAgent.

Solution: If the two passwords are the same, check whether an unsupported parameter is included among the modified options, or whether the name of a parameter is incorrect.

OBD-4001: Fail to send config file to x.x.x.x

Cause: (Check whether the error is caused by either of the reasons.)

• The disk space for the home_path directory on OBAgent is insufficient.

• user in the configuration file (the current user by default, if unspecified) does not have the write permission on the home_path directory on OBAgent.

Solution: You can resolve the problem in two ways.

• Run the following command to add or modify user information:

  obd cluster edit-config <deploy name>
  

• Log on to the target server and grant the current account the write permission on the corresponding directory.

ODP-related errors

OBD-4100: x.x.x.x need config "rs_list" or "obproxy_config_server_url"

Cause: The server cannot obtain the rs_list or obproxy_config_server_url information.

Solution: Run the following command to open the configuration file. Add or modify the rs_list or obproxy_config_server_url parameter and run the restart command provided in the output.

obd cluster edit-config <deploy name>

OBD-4101: failed to start x.x.x.x obproxy: xxx

Cause: Failed to start ODP.

Solution: Perform further analysis based on the error message.

Grafana-related errors

OBD-4200: x.x.x.x grafana admin password should not be 'admin'

Cause: The password of the admin user of Grafana cannot be admin.

Solution: Run the following command to open the configuration file. Add or change the password and run the restart command provided in the output.

obd cluster edit-config <deploy name>

OBD-4201: x.x.x.x grafana admin password length should not be less than 5

Cause: The password of the admin user of Grafana cannot be shorter than 5 characters in length.

Solution: Run the following command to open the configuration file. Add or change the password and run the restart command provided in the output.

obd cluster edit-config <deploy name>

OCP Express-related errors

OBD-4300: x.x.x.x: failed to query java version, you may not have java installed

Cause: OBD cannot obtain the Java information from the server.

Solution:

1. Install Java. For more information, see the How do I configure the Java environment before I deploy OCP Express? section in FAQ.

2. If Java has been installed, set the value of the java_bin parameter to the path of the executable file of Java.

OBD-4301: x.x.x.x: ocp-express need java with version xxx

Cause: The Java version on the server is earlier than needed.

Solution: Install Java of the version provided in the error message. If the target Java version has been installed, set the value of the java_bin parameter to the path of the executable file of Java.

OBD-4302: x.x.x.x not enough memory. (Free: xxx, Need: xxx)

Cause: The server has insufficient memory.

Solution:

• If the total memory available for the server is insufficient, run the obd cluster edit-config command to open the configuration file and modify the memory_limit parameter to a smaller value, or replace the server with one that has sufficient memory.

• If the remaining memory on the server is insufficient, run the following command to release cached objects that can be released.

  echo 3 > /proc/sys/vm/drop_caches
  

OBD-4303: x.x.x.x xxx not enough disk space. (Avail: xxx, Need: xxx)

Cause: The server has insufficient disk space.

Solution: Check and clean up the disk.

OBD-4304: OCP express xxx needs to use xxx with version xxx or above

Cause: OCP Express must be used with the corresponding component of the specified version.

Solution: Run the following command to open the configuration file. Modify the version of the corresponding component and run the restart command provided in the output.

obd cluster edit-config <deploy_name>

OBD-4305: There is not enough xxx for ocp meta tenant

Cause: The log disk space and memory are insufficient for creating a meta tenant for OCP Express.

Solution:

• If the OceanBase cluster is deployed in Maximum Utilization mode on the GUI, or deployed by using the obd cluster autodeploy command on the CLI, we recommend that you clean up the disk and memory and then try again.

• If a resource specification is configured for the cluster, increase the values of the resource-related parameters for the oceanbase-ce component, for example, the memory-related parameters such as memory_limit and memory_limit_percentage and the log disk-related parameters such as log_disk_size and log_disk_percentage.

OBD-4306: xxx ocp-express admin_passwd invalid

Cause: The password is incorrectly configured for the administrator account used to log on to OCP Express.

Solution: Run the obd cluster edit-config command to open the configuration file and modify the admin_passwd parameter. The password can contain letters, digits, and special characters. The password must be 8 to 32 characters in length and contain at least two uppercase letters, two lowercase letters, two digits, and two special characters. The supported special characters are ~!@#%^&*_-+=`|(){}[]:;',.?/.

Config Server-related errors

OBD-4401: Failed to start x.x.x.x ob-configserver

Causes:

1. Cause 1: An internal runtime error occurs when Config Server is started, which leads to service termination.

2. Cause 2: The listening port of Config Server is not enabled in the target deployment server, which leads to access failures.

Solution: Log on to the target deployment server and run the following command:

ps -ef | grep $home_path/bin/ob-configserver

where $home_path specifies the working directory of Config Server. If no running Config Server process is displayed in the returned result, the error is caused by Cause 1. Otherwise, the error is caused by Cause 2.

For Cause 1, you can open the ob-configserver.log file in the $home_path/log directory and view the keyword in the error information. In most cases, the error is caused by incorrect configuration of the connection_url parameter when database_type is set to sqlite3. In this case, correct the configuration error. If the issue persists, submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

For Cause 2, take the following actions:

• If you use a cloud server instance, log on to the instance and add the listening port of Config Server to the allowlist of the instance.

• If you use a self-managed server, enable the listening port based on the operating system version.

OBD-4402: x.x.x.x ob-configserver config error

Cause: An error is detected in the configurations of Config Server.

Solution: Check whether any parameter is missing or incorrectly configured based on the error description. Perform the check as follows:

• If a virtual IP address (VIP) is used, check whether vip_address and vip_port are specified.

• Check whether database_type and connection_url are specified. When database_type is set to sqlite3, connection_url is optional.

• Check whether the database_type parameter is correctly configured. Valid values of database_type are mysql and sqlite3.

• If database_type is set to sqlite3, check whether the value of connection_url is an absolute path.

OBD-4403: x.x.x.x: /xxxx/xxxx/xxxx: permission denied

Cause: When database_type is set to sqlite3 for Config Server, the user specified in the configuration file (or the current user if no user is specified in the configuration file) does not have the write privilege on the directory specified by connection_url.

Solution: You can resolve the issue in one of the following ways:

• Run the following command to open the configuration file and add or modify the user information:

  obd cluster edit-config <deploy name>
  

  After saving the modification, you must restart the cluster by using the command provided in the returned result.

• Log on to the target server and grant the current account the write privilege on the corresponding directory.

OBD-4404: xxxxx: failed to connect to database: xxxx

Cause: When database_type is set to mysql, the database specified by connection_url cannot be connected.

Solution: Verify whether the database specified by connection_url can be connected. If not, specify a connectable database.

SQL-related errors

OBD-5000: sql execute failed

Cause: The SQL execution failed.

Solution: Determine the solution based on the actual situation.

obdiag-related errors

OBD-6000: Failed to executable obdiag command, you may not have obdiag installed

Cause: The obdiag component is not installed.

Solution: Run the following command to install the obdiag component:

obd obdiag deploy

OBD-6001: obdiag must contain depend components xxxx

Cause: The component that obdiag depends on is not installed. The obdiag service of OceanBase Deployer (OBD) depends on an OceanBase cluster or OceanBase Database Proxy (ODP) cluster deployed in OBD. If no OceanBase or ODP cluster is deployed in OBD, this error is returned.

Solution: Install the component that the obdiag service depends on. Make sure that at least one OceanBase or ODP cluster is registered with OBD. You can run the obd cluster list command to view all clusters registered with OBD.

OBD-6002: obdiag options xxx format error, please check the value : xxx

Cause: The format of a parameter value in an obdiag command is incorrect.

Solution: Append the -h option to the obdiag command to view the parameter requirements, and specify parameters in correct formats.

# example
obd obdiag gather -h

obd obdiag gather log -h

Unexpected errors

OBD-9999: Unexpected exception: need to be posted on "https://ask.oceanbase.com", and we will help you resolve them

Cause: An unexpected exception occurs during the operation.

Solution: Submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.