Error codes

This topic summarizes the errors that may occur during the use of OceanBase Deployer (obd).

General errors

OBD-1000: Configuration conflict x.x.x.x: xxx port is used for x.x.x.x

Cause: Port conflicts occur in the configuration file.

Solution: The solution varies based on the deployment mode.

• For command line-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, modify the port settings, save the configuration file, and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page (during OceanBase Database deployment) or MetaDB configuration page (during OCP deployment), find the port mentioned in the error message and change it.

OBD-1001: x.x.x.x:xxx port is already used

Cause: The port has been occupied.

Solution: You can choose to terminate the process that occupies this port or use a port that is not occupied. You can change the port by using any of the following methods:

• Method 1: If the cluster is deployed by using a configuration file, you can run the obd cluster edit-config <deploy name> command to open the configuration file. Then, change the port in the configuration file and run the obd cluster start to start the cluster.

  Note

  For more information about the commands used in Method 1, see Cluster commands.

• Method 2: If the cluster is deployed by using the obd demo command, run the following command to specify the port. In this example, the mysql_port parameter of the oceanbase-ce component is specified:

  obd demo --oceanbase-ce.mysql_port=3881
  

  Note

  For more information about the commands used in Method 2, see Quick deployment command.

• Method 3: If the cluster is deployed by using the obd GUI, you can click Previous, find the port mentioned in the error message, and change it to a port that is not occupied.

OBD-1002: Fail to init x.x.x.x path

Causes: There exists two possible causes. You can determine the specific cause based on the error message.

1. The user specified by user in the configuration file, or the current user by default if user in the configuration file is not specified, does not have the write privilege on the corresponding directory.

2. The specified directory is not empty.

Solution:

For the first case, you can resolve the issue in two ways.

Run the obd cluster edit-config <deploy name> command to open the configuration file. Then, add or modify the user information in the configuration file and run the command provided in the output to put the modification into effect.

• Log on to the target server and grant the current account the write privilege on the corresponding directory.

For the second case, you can also resolve the issue in two ways.

• Run the obd cluster edit-config <deploy name> command to open the configuration file. Then, change the value of the corresponding parameter to an empty directory.

• If you are sure that the current directory can be cleared, you can add the -f option to the command and run the command again. OBD will clear the directory based on the privileges of the current user.

OBD-1003: fail to clean x.x.x.x:xxx

Cause: The user specified by user in the configuration file, or the current user by default if user in the configuration file is not specified, does not have the write privilege on the directory specified by home_path.

Solution: You can resolve the issue in two ways.

• Run the obd cluster edit-config <deploy name> command to open the configuration file. Then, add or modify the user information in the configuration file and run the command provided in the output to put the modification into effect.

• Log on to the target server and grant the current account the write privilege on the corresponding directory.

OBD-1004: Configuration conflict x.x.x.x: xxx is used for x.x.x.x

Cause: Path conflicts occur in the configuration file.

Solution: Check and modify the configuration.

OBD-1005: Some of the servers in the cluster have been stopped

Cause: Some servers in the current configuration have been stopped, but subsequent operations require the services of all servers to be online.

Solution: Run the obd cluster start <deploy_name> --wop command to start all services without loading parameters.

OBD-1006: Failed to connect to xxx

Cause:

1. obd and the specified server are disconnected.

2. The corresponding component process has exited or does not provide services.

3. The account and password do not match.

Solution:

For the first case, you need to solve the network connection issue.

For the second case, you can try to start the component again. If the component fails to be started, troubleshoot the issue based on the error message corresponding to the error code, such as OBD-2002, indicating a startup failure.

For the third case, you may have directly executed an SQL statement to change the password to one that is different from that stored in the configuration file. As a result, obd cannot connect to the component. In this case, you can use either of the following two solutions:

1. Execute an SQL statement to change the password back to the one stored in the configuration file of obd.

2. Run the vi ~/.obd/cluster/<deploy name>/config.yaml command to change the password to the one that is in use for the component.

OBD-1007: (x.x.x.x) The value of the ulimit parameter xxx must not be less than xxx (Current value: xxx)

Cause: The value of the ulimit parameter does not meet the requirements.

Solution: Modify the corresponding file in the /etc/security/limits.d/ directory and the limits.conf file in the /etc/security directory to make sure that the parameter value meets the requirements.

OBD-1008: (x.x.x.x) failed to get fs.aio-max-nr and fs.aio-nr

Cause: obd cannot obtain the asynchronous I/O (AIO) configuration from the server.

Solution: Check whether the current user has the privilege to view fs.aio-max-nr/fs.aio-nr.

cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr

OBD-1009: x.x.x.x xxx need config: xxx

Cause: A parameter is missing for the related service component.

Solution: Run the obd cluster edit-config <deploy name> command to open the configuration file. Then, add the missing parameter to the configuration file and run the command provided in the output to put the modification into effect.

OBD-1010: x.x.x.x No such net interface: xxx

Cause: obd cannot obtain the devname parameter.

Solution: The solution varies based on the deployment mode.

• For command line-based deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, add or modify the devname parameter in the configuration file and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page (during OceanBase Database deployment) or MetaDB configuration page (during OCP deployment), enable More Configurations in the Cluster Configuration section and set the devname parameter.

OBD-1011: (x.x.x.x) Insufficient AIO remaining (Avail: xxx, Need: xxx), The recommended value of fs.aio-max-nr is 1048576

Cause: The AIO resources available in the system are insufficient.

Solution: Run the following command to modify linux aio-max-nr:

sudo sysctl fs.aio-max-nr=1048576

OBD-1012: xxx

Cause:

1. A type conversion exception occurs. For example, a string is passed in for an int parameter.

2. The value of a parameter is beyond the allowed range. For example, the value range of rpc_port is 1025 to 65535. If the value configured for rpc_port is not within this range, an error is returned.

3. A parameter is missing. For example, a key parameter such as home_path is not configured.

Solution:

For the first case, check the parameter type and modify the value.

For the second case, check the passed-in parameter value and modify it.

For the third case, check the parameter configuration. If any parameter is missing, configure it.

OBD-1013: xxx@x.x.x.x connect failed: xxx

Causes (two most common causes):

1. The username or password was incorrect.

2. The connection timed out.

Solution: If the following error message is returned, run the obd env set OBD_DISABLE_RSA_ALGORITHMS 1 command to disable the rsa-sha2-512 and rsa-sha2-256 algorithms:

Open ssh connection \Exception (client): Unable to agree on a pubkey algorithm for signing a 'ssh-rsa' key!

If the preceding error message is not returned, run an SSH command to attempt to connect to the server.

• If the connection fails, verify the connection information and server configurations.

• If the connection succeeds, modify the connection information in either of the following ways based on the deployment mode:

  • For CLI-based deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, add or modify the user information in the configuration file and run the command provided in the output to put the modification into effect.

  • For GUI-based deployment, click Previous. On the Node Configuration page (during OceanBase Database deployment) or MetaDB Configuration page (during OCP deployment), modify the information in the User section.

If the issue persists, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-1015: Unable to confirm the primary-standby relationship, rerun with "--ignore-standby" option if you want to proceed despite the risks

Cause: The cluster or tenant involved in the current operation was engaged in a primary/standby relationship. However, an exception occurred in verifying the relationship when a command was being executed.

Solution: Check whether the current cluster or tenant has a standby tenant in another cluster. The check method varies based on whether the cluster is available.

• If the current cluster is available, you can run the following command to check whether a standby tenant exists. Here a cluster named test is used as an example.

  obd cluster tenant show test -g
  

• If the current tenant is unavailable, you can run the following command to view the primary/standby relationships of the current cluster or tenant. Here a cluster named test is used as an example.

  cat ~/.obd/cluster/test/inner_config.yaml
  

  Based on the returned result, run the following command on the corresponding cluster to check whether the primary/standby relationships still exist. Here a cluster named test-standby is used as an example.

  obd cluster tenant show test-standby -g
  

Take the following actions based on the check result. For more information about the commands involved in the following actions, see Cluster commands.

• If the current operation is an upgrade operation and the current cluster or tenant has a standby tenant, you can upgrade the cluster of the standby tenant first and then that of the primary tenant. This is because the cluster version of the standby tenant must not be earlier than that of the primary tenant. If the current cluster contains both primary and standby tenants, you can first run the obd cluster tenant switchover command to switch the primary tenant in the cluster to a standby tenant. After the clusters where the primary and standby tenants reside are upgraded, you can perform a switchover to switch the standby tenant back to the primary tenant.

  Note

  If the current cluster is undergoing an intra-version upgrade, you can run the command again with the check skipped by appending the --ignore-standby option to the command.

• If the current operation is not an upgrade operation, such as destroy, redeploy, or drop, and the current cluster or tenant has a standby tenant, you can revoke the primary/standby relationship in one of the following ways:

  • Run the obd cluster tenant decouple command to decouple the standby tenant from the primary tenant. After the decoupling, the standby tenant becomes an independent normal tenant.

  • Run the obd cluster stop command to stop the cluster where the primary tenant resides and then run the obd cluster tenant failover command to perform a failover for the standby tenant. After the failover, the standby tenant becomes the new primary tenant.

  • Run the obd cluster tenant drop command to drop the standby tenant.

• If the current cluster or tenant has no standby tenant or if you can accept the risks caused by the unavailability of the standby tenant, you can run the command again with the check skipped by appending the --ignore-standby option to the command.

OBD-1016: (xx.xx.xx.xx) failed to get xxx using command "sysctl -a"

Cause: A connection error occurred or the operating system does not support the sysctl -a command.

Solution: Retry the operation or submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-1017: (xx.xx.xx.xx) The value of the "xxx" must be xxx

Cause: The value of a kernel parameter of the operating system is not within the recommended range.

To ensure the stability of OceanBase Database in a production environment, obd will check the system environment and kernel parameters before OceanBase Database is started. This is to ensure that the system configurations meet the recommended parameter settings of OceanBase Database. If production_mode is set to true or the --strict-check option is specified when you run the command, your instance will be identified as a production environment. In this case, if any parameter is beyond the recommended value range, an error is returned and the startup is terminated. If production_mode is not set to true or the --strict-check option is not specified in the command, only an alert will be triggered without terminating the startup process.

Solution: Use one of the following two methods based on your environment:

• In the production environment, run the sysctl -w {kernel parameter name}="recommended value" command or the echo "kernel parameter name=recommended value" >> /etc/sysctl.conf; sysctl -p command to set the parameter to a recommended value.

• In the test environment and if you are not authorized to modify a kernel parameter, run the obd cluster edit-config {deployname} command to modify the configuration file and set the production_mode parameter to false to skip parameter verification.

OBD-1018: could not change xxx

Cause: The configuration file specified by the command contains a parameter that cannot be modified.

Solution: Check the specified configuration file for parameters such user, depends, unuse_lib_repository, and auto_create_tenant.

OBD-1019: component xxx already in cluster

Cause: The component to be added already exists in the cluster. obd cannot add an existing component to the cluster.

Solution: Run the obd cluster edit-config command to open the configuration file, and view the component information to check whether the component to be added exists in the cluster.

OBD-1020: Update config for component xxx failed

Cause: Failed to update the component configuration.

Solution: Check the configuration file based on the output error code. If you cannot find the root causes, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-1021: Component xxx is not in cluster

Cause: The component to be deleted does not exist in the cluster.

Solution: Run the obd cluster edit-config command to open the configuration file, and view the component information to check whether the component to be deleted exists in the cluster.

OBD-1022: Component xxx still depends by xxx, could not remove

Cause: Another component in the cluster depends on the component to be deleted. obd cannot delete a component that has dependencies.

Solution: Run the obd cluster edit-config command to open the configuration file, and view the dependencies of the component, which are specified by the depends parameter.

OBD-1023: Failed to merge config: xxx

Cause: Failed to merge configuration files.

Solution: Check the configuration file based on the output error code. If you cannot find the root causes, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-1024: The cluster will have no remaining components

Cause: The component to be deleted is the only component in the cluster.

Solution: If you want to delete all components of the cluster, run the obd cluster destroy command to destroy the cluster.

OBD-1025: ({ip}) {component} {key} invalid

Cause: The password configured for the corresponding component does not meet the requirements. obd will check the password based on the password conventions of the component.

• The password of the admin user (admin_password) for logging on to the OceanBase Cloud Platform (OCP) console must meet the following requirements:

  • It must be 8 to 32 characters in length.

  • It must contain characters of at least three of the following types: digits (0 to 9), uppercase letters (A to Z), lowercase letters (a to z), and special characters. Special characters are ~!@#%^&*_-+=|(){}[]:;,.?/$`'"<>\

• The password of the admin user (admin_passwd) for logging on to the OCP Express console must meet the following requirements:

  • It must be 8 to 32 characters in length.

  • It must contain characters of at least three of the following types: digits (0 to 9), uppercase letters (A to Z), lowercase letters (a to z), and special characters. Special characters are ~!@#%^&*_-+=|(){}[]:;,.?/

  Notice

  After you change the password of the admin user (admin_passwd) for logging on to the OCP Express console, you must run the obd cluster redeploy command. The obd cluster redeploy command will destroy the cluster and redeploy it. The data in the cluster will be lost during this process. Back up the data in advance.

• The password for HTTP service authentication (http_basic_auth_password) in OceanBase Agent (OBAgent) must meet the following requirements:

  • The length of the password is [0, +∞).

  • It can contain digits (0 to 9), uppercase letters (A to Z), lowercase letters (a to z), and special characters. Special characters are ~^*{}[]_-+

Solution: For command line-based deployment, you can run the obd cluster edit-config command to open the configuration file. Then, modify the corresponding parameter of the component in the configuration file, save the configuration file, and run the restart command provided in the output.

OBD-1026: Could not modify {key} when the cluster is in the working status(production_mode is True, not support this operation)

Cause: After you modify this parameter, you must run the obd cluster redeploy command to destroy and redeploy the cluster. The modification takes effect after you redeploy the cluster. The data in the cluster will be lost. Therefore, when the production_mode parameter is set to true, you cannot modify this parameter.

Solution: When you deploy an OceanBase cluster, the production_mode parameter is set to true by default. If you do not need to use the cluster in a production environment and can afford the risk of data loss, run the obd cluster edit-config command to set the production_mode parameter to false. Then, restart the cluster by using the command provided in the output and attempt to modify this parameter again.

OBD-1027: If you are sure the directory can be emptied, run obd cluster deploy -f {deploy_name} to perform forced deployment

Cause: The configuration file contains a non-empty directory.

Solution: Clear the directory or specify a new empty directory. To do so, perform the following steps:

• Clear the directory: Manually clear the directory returned in the OBD-1002 error message. You can also copy the command returned in the error message, add the -f option to the command, and run it again. OBD will automatically clear the directory.

• Specify a new empty directory: Run the obd cluster edit-config <deploy name> command to open the configuration file. Then, change the value of the corresponding parameter to a new directory. The parameter is returned in the OBD-1002 error message.

OceanBase Database deployment errors

OBD-2000: x.x.x.x not enough memory

Cause: The available memory is insufficient.

Solution: When obd starts, the memory is strictly calculated based on MemAvailable. If any cached memory can be released, run the following command:

sudo sysctl -w vm.drop_caches=1
  # or
  sudo echo 1 > /proc/sys/vm/drop_caches

If the memory is still insufficient, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, adjust the values of memory_limt and system_memory in the configuration file. Generally, memory_limt/3 ≤ system_memory ≤ memory_limt/2.

OBD-2001: server can not migrate in

Cause: The number of available units is smaller than the value of --unit-num.

Solution: Modify the passed value of --unit-num. Run the following command to view the number of available units:

select count(*) num from oceanbase.__all_server where status = 'active' and start_service_time > 0

OBD-2002: failed to start x.x.x.x observer

Cause (two most common causes):

• The value of memory_limit is less than 8 GB.

• The value of system_memory is too large or small. Generally, the following condition must be met: memory_limt/3 ≤ system_memory ≤ memory_limt/2.

Solution:

• If the issue is caused by either of the preceding reasons, take actions accordingly.

• If the issue persists, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-2003: not enough disk space for clog. Use redo_dir to set other disk for clog, or reduce the value of datafile_size

Cause: The disk usage exceeds the limit.

• For automatic deployment, the disk usage cannot exceed 72%.

• For manual deployment, the disk usage cannot exceed 64%, if the related configurations are not modified.

Solution: Adjust the disk configurations. The adjustment method varies based on the deployment mode.

• For command line-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, modify the disk parameters such as datafile_size, memory_limit, and log_disk_size, save the configuration file, and run the command provided in the output to put the modifications into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page (during OceanBase Database deployment) or MetaDB configuration page (during OCP deployment), enable More Configurations in the Cluster Configuration section and modify the disk parameters such as datafile_size, memory_limit, and log_disk_size.

OBD-2004: Invalid: xxx is not a single server configuration item

Cause: The modified parameter is a global one and cannot be separately modified for a single server.

Solution: Place the parameter to modify under global.

OBD-2005: Failed to register cluster. xxx may have been registered in xxx

Cause: The cluster registration failed or the cluster has been registered.

Solution: Check whether the appname parameter is configured for the cluster. If no, you cannot register the cluster with obconfigserver. Take the following actions based on whether the cluster has been deployed.

• Case 1: If you want to register a to-be-deployed OceanBase cluster with obconfigserver, comment out the obconfig_url parameter, start the cluster, and then run the obd cluster edit-config command to configure the obconfig_url parameter. At present, you cannot register a to-be-deployed cluster with obconfigserver.

• Case 2: If you want to register a started cluster with obconfigserver, check whether the obconfig_url parameter is correctly configured.

  • If obconfig_url is incorrectly configured, run the obd cluster edit-config command to open the configuration file and specify the correct Config URL for the obconfig_url parameter.

  • If obconfig_url is correctly configured and you want to forcibly overwrite the registered cluster, you can append the -f option to the obd cluster start command.

OBD-2006: x.x.x.x has more than one network interface. Please set devname for x.x.x.x

Cause: The server has multiple network interfaces and obd cannot obtain the devname parameter.

Solution: The solution varies based on the deployment mode.

• For command line-based deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, add or modify the devname parameter in the configuration file and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page (during OceanBase Database deployment) or MetaDB configuration page (during OCP deployment), enable More Configurations in the Cluster Configuration section and set the devname parameter.

OBD-2007: x.x.x.x xxx fail to ping x.x.x.x

Cause: The nodes cannot ping each other.

Solution:

1. Check whether the network connection between the nodes is normal.

2. Check whether the network configuration (devname/local_ip) conforms to the actual situation. You can run the ip addr command to view the mappings between IP addresses and network interface cards (NICs). If the network configuration does not conform to the actual situation, modify the configuration. The modification method varies based on the deployment mode.

   • For CLI-based deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, add or modify the devname parameter in the configuration file and run the command provided in the output to put the modification into effect.

   • For GUI-based deployment, click Previous. On the Cluster Configuration page (during OceanBase Database deployment) or MetaDB Configuration page (during OCP deployment), enable More Configurations in the Cluster Configuration section and set the devname parameter.

3. If the operation not permitted error occurs, check the privileges of the ping file . You can run the sudo chmod u+s /usr/bin/ping command to modify the privileges of the ping file.

4. If the No such file or directory error occurs, the ping command is not supported in your environment. In this case, run the sudo yum install iputils or sudo apt-get install iputils-ping command to install the ping command tool.

OBD-2008: Cluster clocks are out of sync

Cause: The clocks on the servers in the cluster are out of synchronization.

Solution: Synchronize the clocks on the servers.

OBD-2009: x.x.x.x: when production_mode is True, xxx can not be less then xxx

Cause: In the production mode, the values of parameters such as __min_full_resource_pool_mem and memory_limit cannot be smaller than specified values.

Solution:

• In a non-production environment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, change the value of the production_mode parameter in the configuration file to False and run the command provided in the output to put the modification into effect.

• In a production environment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, modify the __min_full_resource_pool_mem and memory_limit parameters to values larger than the specified values, and run the command provided in the output to put the modifications into effect.

OBD-2010: x.x.x.x: system_memory too large. system_memory must be less than memory_limit/memory_limit_percentage

Cause: The value of system_memory must be smaller than the value of memory_limit or memory_limit_percentage × total_memory.

Solution: The solution varies based on the deployment mode.

• For command line-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, change the value of the system_memory parameter, save the configuration file, and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page (during OceanBase Database deployment) or MetaDB configuration page (during OCP deployment) page, enable More Configurations in the Cluster Configuration section and set system_memory.

OBD-2011: x.x.x.x: fail to get memory info.\nPlease configure 'memory_limit' manually in configuration file

Cause: The server cannot obtain the memory information.

Solution: The solution varies based on the deployment mode.

• For command line-based deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, set memory_limit, save the configuration file, and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page (during OceanBase Database deployment) or MetaDB configuration page (during OCP deployment) page, enable More Configurations in the Cluster Configuration section and set memory_limit.

Testing-related errors

OBD-3000: parse cmd failed

Cause: The mysqltest initialization file is not an .sql file.

Solution: Check the --init-sql-files parameter.

OBD-3001: xxx.sql not found

Cause: The initialization file cannot be found during the initialization of mysqltest.

Solution: Check whether the file declared by --init-sql-files is located in the --init-sql-dir directory.

OBD-3002: Failed to load data

Cause (two most common causes):

1. The tenant has insufficient resources or is under excessive stress.

2. An error occurred in the data build script.

Solution:

For the first case, you can use a tenant with larger resource specifications or adjust parameters such as warehouses and load-workers to reduce the test stress.

For the second case, try to re-execute the data build script. If the issue persists, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-3003: Failed to run TPC-C benchmark

Cause:

1. The test process was stuck and then terminated due to timeout.

2. An error is returned for the TPC-C test command.

Solution:

• Run the test again directly or after adjusting parameters such as terminals to reduce the test stress.

• If you did not use the obtpcc package provided on the OceanBase Database official website, use obtpcc for testing.

If the issue persists, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBAgent-related errors

OBD-4000: Fail to reload x.x.x.x

Cause: The http_basic_auth_password setting of the node is not the same as that stored in obd. As a result, obd failed to access OBAgent.

Solution: If the two passwords are the same, check whether an unsupported parameter or an incorrect parameter name is included among the modified options.

OBD-4001: Fail to send config file to x.x.x.x

Cause: (Check whether the error is caused by either of the reasons.)

• The disk space for the directory specified by home_path on OBAgent is insufficient.

• The user specified by user in the configuration file, or the current user by default if user in the configuration file is not specified, does not have the write privilege on the directory specified by home_path on OBAgent.

Solution: You can resolve the issue in two ways.

• Run the obd cluster edit-config <deploy_name> command to add or modify the user information, and save the configuration file. Then, run the command provided in the output to put the modification into effect.

• Log on to the target server and grant the current account the write privilege on the corresponding directory.

OBD-4002: {servers}: Failed to obtain the configuration of the OceanBase database component

Cause: The servers information of OBAgent in the configuration file is inconsistent with that of OceanBase Database. Therefore, OBAgent cannot obtain configurations of OceanBase Database.

Solution: Run theobd cluster edit-config command to modify the servers information of OBAgent or OceanBase Database in the configuration file. Make sure that the servers information of the two components is consistent.

ODP-related errors

OBD-4100: x.x.x.x need config "rs_list" or "obproxy_config_server_url"

Cause: The server cannot obtain the rs_list or obproxy_config_server_url information.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file. Add or modify the rs_list or obproxy_config_server_url parameter, save the configuration file, and run the command provided in the output to put the modification into effect.

OBD-4101: failed to start x.x.x.x obproxy: xxx

Cause: Failed to start ODP.

Solution: Perform further analysis based on the error message.

ODP-4102: When the value of client_session_id_version is set to xx, the valid range of proxy_id is xxxx

Cause: The value of the proxy_id parameter is beyond the valid value range. The valid value range of the proxy_id parameter varies based on the value of the client_session_id_version parameter.

• When the value of client_session_id_version is 1, the valid value range of proxy_id is [1,255].

• When the value of client_session_id_version is 2, the valid value range of proxy_id is [1,8191].

Solution: Run the obd cluster edit-config command to open the configuration file. Then, change the value of the proxy_id parameter, save the configuration file, and run the restart command provided in the output to restart the cluster.

Grafana-related errors

OBD-4200: x.x.x.x grafana admin password should not be 'admin'

Cause: The password of the admin user of Grafana cannot be admin.

Solution: Run the following command to open the configuration file. Add or change the password and run the restart command provided in the output.

obd cluster edit-config <deploy name>

OBD-4201: x.x.x.x grafana admin password length should not be less than 5

Cause: The password of the admin user of Grafana cannot be shorter than 5 characters in length.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, add or modify the password information, save the configuration file, and run the command provided in the output to put the modification into effect.

OCP Express-related errors

OBD-4300: x.x.x.x: failed to query java version, you may not have java installed

Cause: obd cannot obtain the Java information from the server.

Solution:

1. Install Java. For more information, see the How do I configure the Java environment before I deploy OCP Express? section in FAQ.

2. If Java has been installed, set the value of the java_bin parameter to the path of the executable file of Java. The configuration method varies based on the deployment mode.

   • For command line-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, change the value of the java_bin parameter to the path where the executable file of Java is located, save the configuration file, and run the command provided in the output to put the modification into effect.

   • For GUI-based deployment, click Previous. On the Cluster Configuration page, enable More Configurations in the Component Configuration section and change the value of java_bin to the path where the executable file of Java is located.

OBD-4301: x.x.x.x: ocp-express need java with version xxx

Cause: The Java version on the server is earlier than needed.

Solution: Install Java of the prompted version. If the version has been installed, set the value of the java_bin parameter to the path where the executable file of Java is located. For more information, see OBD-4300.

OBD-4302: x.x.x.x not enough memory. (Free: xxx, Need: xxx)

Cause: The server has insufficient memory.

Solution:

• If the total memory of the server is insufficient, decrease the value of the memory_size parameter or replace the server with one that has sufficient memory. The method for decreasing the value of the memory_size parameter varies based on the deployment mode.

  • For command line-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, decrease the value of the memory_size parameter, save the configuration file, and run the command provided in the output to put the modification into effect.

  • For GUI-based deployment, click Previous. On the Cluster Configuration page, enable More Configurations in the Cluster Configuration section and decrease the value of the memory_size parameter.

• If the remaining memory on the server is insufficient, run the following command to release cached objects that can be released.

  sudo sysctl -w vm.drop_caches=1
  # or
  sudo echo 1 > /proc/sys/vm/drop_caches
  

OBD-4303: x.x.x.x xxx not enough disk space. (Avail: xxx, Need: xxx)

Cause: The server has insufficient disk space.

Solution: Check and clean up the disk, or decrease the value of the logging_file_total_size_cap parameter. The method for decreasing the value of the logging_file_total_size_cap parameter varies based on the deployment mode.

• For command line-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, decrease the value of the logging_file_total_size_cap parameter, save the configuration file, and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page, enable More Configurations in the Component Configuration section and decrease the value of the logging_file_total_size_cap parameter

OBD-4304: OCP express xxx needs to use xxx with version xxx or above

Cause: OCP Express must be used with the corresponding component of the specified version.

Solution: Run the obd cluster edit-config <deploy name> command to open the configuration file. Then, modify the version of the component mentioned in the error message, save the configuration file, and run the command provided in the output to put the modification into effect.

OBD-4305: There is not enough xxx for ocp meta tenant

Cause: The log disk space and memory are insufficient for creating a meta tenant for OCP Express.

Solution: Clean up the disk and memory and then try again, or modify related parameters as follows based on the deployment mode:

• For command line-based deployment, if cluster specifications are configured, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, increase the values of parameters of the oceanbase-ce component, such as the memory parameters memory_limit and memory_limit_percentage and log disk parameters log_disk_size and log_disk_percentage, or decrease the value of the memory_size parameter of ocp_meta_tenant. Save the configuration file, and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page, enable More Configurations in the Cluster Configuration section and decrease the value of the ocp_meta_tenant_memory_size parameter or increase the values of the memory_limit and log_disk_size parameters.

OCP-related errors

OBD-4350: The Server have running task

Cause: When you upgrade OCP, OCP is running a task.

Solution: Upgrade OCP after the task is completed.

OBD-4351: The Server have gone

Cause: When you upgrade OCP, a host managed by OCP is not online.

Solution: Query the status of the hosts managed by OCP. For more information, see Host management operations in the OCP documentation.

• If a host is in the Newly Submitted state, it is a new host. Upgrade OCP after the host is added.

• If a host is in the Offline state, reinstall OCP Agent. For more information, see Reinstall OCP Agent.

OBD-4352: Metadb version not fewer than V2.2.50

Cause: The version of OCP MetaDB is earlier than V2.2.50.

Solution: Upgrade OCP MetaDB to the latest Long-Term Support (LTS) version.

OBD-4353: {server}: Excessive deviation between machine time and ob time

Cause: The time deviation between the host time and the MetaDB time is too large.

Solution: Perform the following steps.

1. Run the following commands on the corresponding host to check whether the Chrony or Network Time Protocol (NTP) clock synchronization service is installed on the server:

   rpm -qa | grep chrony   # Check whether the Chrony service is installed.
   rpm -qa | grep ntp      # Check whether the NTP service is installed.
   

   Use one of the following two methods based on the check result:

   • If the version of the Chrony or NTP service is returned, the corresponding service is installed. In this case, proceed to Step 2.

   • If no version information is returned, the corresponding service is not installed. If neither Chrony nor NTP is installed, install a clock synchronization service first. For more information about the installation and configuration of the Chrony and NTP services, see examples shared on the Internet. This topic provides only a brief introduction to the installation and configuration of the Chrony and NTP services.

     • Run one of the following commands to install a clock synchronization service (Chrony or NTP):

       sudo yum install -y chrony    # Install the Chrony service.
       sudo yum install -y ntp       # Install the NTP service.
       

     • Run one of the following commands to start the clock synchronization service:

       systemctl start chronyd     # Start the Chrony service.
       systemctl start ntpd        # Start the NTP service.
       

     • Perform a precheck again. If the issue persists, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

2. Run one of the following commands to check whether the clock synchronization process (chronyd or ntpd) has exited unexpectedly:

   systemctl status chronyd      # Check the status of the Chrony service.
   systemctl status ntpd         # Check the status of the NTP service.
   

   Use one of the following two methods based on the check result:

   • If the value of the Active parameter in the response is active (running), submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

   • If the value of the Active parameter in the response is inactive (dead), the clock synchronization service fails. In this case, run one of the following commands to restart the service:

     systemctl restart chronyd      # Restart the Chrony service.
     systemctl restart ntpd         # Restart the NTP service.
     

     After you restart the service, perform the deployment operation again. If the issue persists, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

OBD-4354: {user}@{server}: Not exist

Cause: The specified OCP startup user does not exist.

Solution: Specify another startup user or create a startup user on the host where OCP resides. For more information about how to create a startup user, see User planning in the OCP documentation.

OBD-4355: {user}@{ip}: user xxx not in sudoers or sudoers file not exist

Cause: The specified user is not authorized to run the sudo command without password.

Solution: Grant the user the permission to run the sudo command without password, or use another user account that has the permission. For more information about how to grant sudo permissions, see User planning in the OCP documentation.

OBD-4356: failed to connect meta db

Cause: MetaDB cannot be connected.

Solution: Check whether the connection string of MetaDB is correct.

OBD-4357: database in jdbc_url is not exist

Cause: The database specified in the JDBC connection string does not exist.

Solution: Create the database in MetaDB.

OBD-4358: unmatched jdbc url, skip meta db connection check

Cause: The format of the JDBC connection string is invalid.

Solution: Make sure that the value of the jdbc_url parameter is in the "^jdbc:\S+://(\S+?)(|:\d+)/(\S+)" format.

OBD-4359: {server}: ocp-server need java with version xxx and update release must greater than 161

Cause: The JDK version does not meet the requirements of OCP.

Solution: Upgrade the version of JDK to 1.8.0_161 or later.

OBD-4360: {server}: clockdiff not exists. Please install clockdiff manually

Cause: The clockdiff program is not installed on the host.

Solution: Install the clockdiff program.

OBD-4361: tenant(xxx) already exist

Cause: The specified tenant already exists.

Solution: Log on to MetaDB and delete the tenant with the same name or use another tenant name.

OBD-4362: {server}:{path} access failed for current user, {server}:{cur_path} access succeed, please run chmod -R 755 {cur_path}

Cause: The specified user is not authorized to operate the specified directory.

Solution: Execute the chmod command in the returned message to grant the user the permissions on the directory.

OBD-4363: ocp server xxx needs to use xxx with version xxx or above

Cause: The component version is not supported by OCP.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, modify the version of the component mentioned in the error message, save the configuration file, and run the command provided in the output to put the modification into effect.

OBD-4364: (xx.xx.xx.xx) not enough memory

Cause: The server has insufficient memory.

Solution:

• If the total memory of the server is insufficient, decrease the value of the memory_size parameter or replace the server with one that has sufficient memory. The method for decreasing the value of the memory_size parameter varies based on the deployment mode.

  • For command line-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, decrease the value of the memory_size parameter and run the command provided in the output for the modification to take effect.

  • For GUI-based deployment, click Previous. On the MetaDB configuration page, enable More Configurations in the Cluster Configuration section and decrease the value of the memory_size parameter.

• If the remaining memory on the server is insufficient, run the following command to release cached objects that can be released.

  sudo sysctl -w vm.drop_caches=1
  # or
  sudo echo 1 > /proc/sys/vm/drop_caches
  

OBD-4365: x.x.x.x xxx not enough disk space. (Avail: xxx, Need: xxx)

Cause: The server has insufficient disk space.

Solution: Check and clean up the disk.

OBD-4366: There is not enough {resource}. (Avail: {avail}, need: {need})

Cause: MetaDB has insufficient memory.

Solution: The solution varies based on the deployment mode.

• For CLI-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, increase the values of the memory_limit, cpu_count, and log_disk_size parameters for the oceanbase-ce component, save the configuration file, and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the MetaDB Configuration page, enable More Configurations in the Cluster Configuration section and increase the values of the memroy_limit, cpu_count, and log_disk_size parameters.

OBD-4367: The allocated memory for the provided meta database is currently insufficient for creating a tenant

Cause: MetaDB has insufficient memory for creating a tenant.

Solution: Increase the memory size of MetaDB or decrease the memory size of the tenant. The solution varies based on the deployment mode.

• For CLI-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, increase the value of the memory_limit parameter for the oceanbase-ce component, or decrease the value of the memory_size parameter for ocp_meta_tenant and ocp_monitor_tenant of the oceanbase-ce component, and save the configuration file. Then, run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the MetaDB Configuration page, enable More Configurations in the Cluster Configuration section and increase the value of the memory_limit parameter. You can also decrease the memory sizes in the MetaDB Tenant and MonitorDB Tenant sections on the OCP Configuration page.

OBD-4368: The allocated memory for the provided meta database is currently insufficient for creating a tenant

Cause: MetaDB has insufficient memory for creating a tenant.

Solution: Decrease the memory size of the tenant. The solution varies based on the deployment mode.

• For CLI-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, decrease the value of the memory_size parameter for ocp_meta_tenant and ocp_monitor_tenant of the ocp-server-ce component, and save the configuration file. Then, run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the OCP Configuration page, decrease the memory sizes in the MetaDB Tenant and MonitorDB Tenant sections.

obconfigserver-related errors

OBD-4401: Failed to start x.x.x.x ob-configserver

Cause:

1. Cause 1: An internal runtime error occurs when obconfigserver is started, which leads to service termination.

2. Cause 2: The listening port of obconfigserver is not enabled in the target deployment server, which leads to access failures.

Solution: Log on to the target deployment server and run the following command:

ps -ef | grep $home_path/bin/ob-configserver

$home_path specifies the working directory of obconfigserver. If no running obconfigserver process is displayed in the returned result, the error is caused by Cause 1. Otherwise, the error is caused by Cause 2.

For Cause 1, you can open the ob-configserver.log file in the $home_path/log directory and view the keyword in the error information. In most cases, the error is caused by incorrect configuration of the connection_url parameter when database_type is set to sqlite3. In this case, correct the configuration error. If the issue persists, you can submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.

For Cause 2, take the following actions:

• If you use a cloud server instance, log on to the instance and add the listening port of obconfigserver to the allowlist of the instance.

• If you use a self-managed server, enable the listening port based on the operating system version.

OBD-4402: x.x.x.x ob-configserver config error

Cause: An error is detected in the configurations of obconfigserver.

Solution: Check whether any parameter is missed or incorrectly configured based on the error description. Perform the check as follows:

• If a virtual IP address (VIP) is used, check whether vip_address and vip_port are specified.

• Check whether database_type and connection_url are specified. When database_type is set to sqlite3, connection_url is optional.

• Check whether the database_type parameter is correctly configured. Valid values of database_type are mysql and sqlite3.

• If database_type is set to sqlite3, check whether the value of connection_url is an absolute path.

OBD-4403: ob-configserver connect to sqlite failed: x.x.x.x: /xxx/xxx/xxx: permission denied

Cause: When database_type is set to sqlite3 for obconfigserver, the user specified in the configuration file (or the current user, if no user is specified in the configuration file) does not have the write privilege on the directory specified by connection_url.

Solution: You can resolve the issue in one of the following ways:

• Run the obd cluster edit-config <deploy_name> command to open the configuration file. Add or modify the user information, save the configuration file, and run the command provided in the output to put the modification into effect.

• Log on to the target server and grant the current account the write privilege on the corresponding directory.

OBD-4404: ob-configserver connect to mysql failed: xxx: failed url to connect to database: xxx

Cause: When database_type is set to mysql, the database specified by connection_url cannot be connected to.

Solution: Verify whether the database specified by connection_url can be connected to. If no, specify a connectable database.

OBD-4405: When you configure multiple ob-configserver servers, please set vip_address and vip_port

Cause: No virtual IP address (VIP) was configured when you deployed obconfigserver on multiple nodes.

Solution: Configure load balancing before you deploy obconfigserver on multiple nodes and specify the vip_address and vip_port parameters during the deployment. You can specify the parameters in either of the following ways based on the deployment mode:

• For CLI-based deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Then, specify the vip_address and vip_port parameters in the configuration file and run the command provided in the output to put the modification into effect.

• For GUI-based deployment, click Previous. On the Cluster Configuration page, enable More Configurations in the Component Configuration section and specify the vip_address and vip_port parameters.

oblogproxy-related errors

OBD-4501: oblogproxy xxx needs to use xxx with version xxx or above

Cause: A component of an incorrect version was used during the deployment of oblogproxy.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file. Then, modify the version of the component mentioned in the error message, save the configuration file, and run the command provided in the output to put the modification into effect.

SQL-related errors

OBD-5000: sql execute failed

Cause: The SQL execution failed.

Solution: Determine the solution based on the actual situation.

obdiag errors

OBD-6000: Failed to executable obdiag command, you may not have obdiag installed

Cause: The obdiag component is not installed.

Solution: Run the following command to install the obdiag component:

obd obdiag deploy

OBD-6001: obdiag must contain depend components xxxx

Cause: The component that obdiag depends on is not installed. The obdiag service of OceanBase Deployer (obd) depends on an OceanBase cluster or OceanBase Database Proxy (ODP) cluster deployed in obd. If no OceanBase or ODP cluster is deployed in obd, this error is returned.

Solution: Install the component that the obdiag service depends on. Make sure that at least one OceanBase or ODP cluster is registered with obd. You can run the obd cluster list command to view all clusters registered with obd.

OBD-6002: obdiag options xxx format error, please check the value : xxx

Cause: The format of a parameter value in an obdiag command is incorrect.

Solution: Append the -h option to the obdiag command to view the parameter requirements, and specify parameters in correct formats.

# example
obd obdiag gather -h

obd obdiag gather log -h

OBD-6003: Failed to execute obdiag function xxx

Cause: obdiag failed to run the command. It is likely that the obdiag command contains options that do not exist.

Solution: Run the following command to view the obdiag commands supported by obd.

obd obdiag -h

Unexpected errors

OBD-9999: Unexpected exception: need to be posted on "https://ask.oceanbase.com", and we will help you resolve them

Cause: An unexpected exception occurs during the operation.

Solution: Submit an issue on GitHub (https://github.com/oceanbase/obdeploy/issues), and designated professionals will help you fix the issue.