Error codes

This topic summarizes the error codes that may occur during the use of obd, mainly including the following aspects.

General error

OBD-1000: Configuration conflict x.x.x.x: xxx port is used for x.x.x.x

Cause: A port conflict exists in the configuration file.

Solution: You can use the following two methods based on the deployment method.

• If you deploy the cluster by using the CLI, run the obd cluster edit-config <deploy name> command to open the configuration file, check the port configuration, and modify it. After the modification is saved, run the command output in the CLI to make the modification take effect.

• If you deploy the cluster by using the GUI, click Previous to go back to the Cluster Configuration (when you deploy OceanBase Database) or MetaDB Configuration (when you deploy OCP) page. On the page, find the port in the error message and modify it.

OBD-1001: x.x.x.x:xxx port is already used

Cause: The port is occupied.

Solution: You can end the process that occupies the port or change the port to an unoccupied one. You can choose one of the following methods based on your needs.

• Method 1: If you deploy the cluster by using a configuration file, run the obd cluster edit-config <deploy name> command to open the configuration file and modify the port configuration in the file. Then, run the obd cluster start command to start the cluster.

  Note

  For more information about the commands mentioned in Method 1, see Cluster commands.

• Method 2: If you deploy the cluster by using the obd demo command, you can specify the port by using the following command. In this example, the mysql_port of the oceanbase-ce component is specified.

  obd demo --oceanbase-ce.mysql_port=3881
  

  Note

  For more information about the commands mentioned in Method 2, see Quick deployment commands.

• Method 3: If you deploy the cluster by using the GUI, click Previous until you find the port in the error message and change it to an unoccupied one.

OBD-1002: Fail to init x.x.x.x path

Cause: The following two reasons may cause this error. You can determine the cause based on the specific error message.

1. The user specified in the configuration file (the current user by default if no user is specified) does not have write permissions for the directory.

2. The specified directory is not empty.

Solution:

For case 1, you can use the following two methods to resolve the issue.

• Run the obd cluster edit-config <deploy name> command to open the configuration file, add or modify the user information in the file, and then run the command output in the CLI to make the modification take effect.

• Log in to the target server and grant write permissions for the directory to the current account.

For case 2, you can use the following two methods to resolve the issue.

• Run the obd cluster edit-config <deploy name> command to open the configuration file and change the value of the corresponding configuration item to an empty directory.

• If you confirm that the directory can be cleared, you can re-execute the command and add the -f option. obd will then use the current user to clear the directory.

OBD-1003: fail to clean x.x.x.x:xxx

Cause: The user specified in the configuration file (the current user by default if no user is specified) does not have write permissions for the home_path.

Solution: You can use the following two methods to resolve the issue.

• Run the obd cluster edit-config <deploy name> command to open the configuration file, add or modify the user information in the file, and then run the command output in the CLI to make the modification take effect.

• Log in to the target server and grant write permissions for the directory to the current account.

OBD-1004: Configuration conflict x.x.x.x: xxx is used for x.x.x.x

Cause: A path conflict exists in the configuration file.

Solution: Check the configuration and modify it.

OBD-1005: Some of the servers in the cluster have been stopped

Cause: All servers in the cluster must be online for the subsequent operations. However, some servers in the current configuration have been stopped.

Solution: You can run the obd cluster start <deploy_name> --wop command to start all services.

OBD-1006: Failed to connect to xxx

Cause:

1. The network between obd and the target server is disconnected.

2. The corresponding component process has exited or is not providing services.

3. The username and password do not match.

Solution:

For case 1, fix the network.

For case 2, try to start the component again. If the component still fails to start, refer to the error message for troubleshooting, such as OBD-2002.

For case 3, the most common cause is that the user directly executed an SQL command to change the password, resulting in a mismatch between the username and password in the configuration file and the actual password. In this case, you can use the following two methods to resolve the issue.

1. Execute an SQL command to change the password back to the one stored by obd.

2. Execute the vi ~/.obd/cluster/<deploy name>/config.yaml command to modify the corresponding password in the configuration file to match the actual password of the component.

OBD-1007: (x.x.x.x) The value of the ulimit parameter xxx must not be less than xxx (Current value: xxx)

Cause: The ulimit configuration does not meet the requirements.

Solution: You can modify the corresponding file in the /etc/security/limits.d/ directory and the /etc/security/limits.conf file to meet the requirements.

OBD-1008: (x.x.x.x) failed to get fs.aio-max-nr and fs.aio-nr

Cause: obd cannot obtain the aio configuration of the server.

Solution: Check whether the current user has the permission to view fs.aio-max-nr/fs.aio-nr.

cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr

OBD-1009: x.x.x.x xxx need config: xxx

Cause: The corresponding configuration is missing for the service component.

Solution: Run the obd cluster edit-config <deploy name> command to open the configuration file and add the configuration item indicated in the error message to the file. Then, run the command output in the CLI to make the modification take effect.

OBD-1010: x.x.x.x No such net interface: xxx

The error occurs because the devname cannot be obtained by obd.

To resolve this issue, perform the following operations based on your deployment method.

• If you deploy the cluster in command-line mode, run the obd cluster edit-config <deploy_name> command to open the configuration file. Add or modify the devname parameter in the configuration file. Save the file and run the command that is displayed in the command line to make the modification take effect.

• If you deploy the cluster in GUI mode, click Previous. On the Cluster Configuration page (when you deploy OceanBase Database) or the MetaDB Configuration page (when you deploy OCP), click More Settings in the Cluster Configuration section and set the devname.

OBD-1011: (x.x.x.x) Insufficient AIO remaining (Avail: xxx, Need: xxx), The recommended value of fs.aio-max-nr is 1048576

The error occurs because the number of available AIOs is less than the number of AIOs required by the database.

To resolve this issue, run the following command to modify the aio-max-nr parameter of Linux.

sudo sysctl fs.aio-max-nr=1048576

OBD-1012: xxx

The error occurs for the following reasons:

1. A type conversion error occurs, such as when a string is passed to an integer parameter.

2. The parameter value exceeds the limit. For example, if the value of the rpc_port parameter is out of the range of 1025 to 65535, an error occurs.

3. A critical parameter is missing. For example, the home_path parameter is not configured.

To resolve this issue:

• For the first reason, check the parameter type and modify it.

• For the second reason, check the parameter value and modify it.

• For the third reason, check the parameter configuration. If a parameter is missing, configure it.

OBD-1013: xxx@x.x.x.x connect failed: xxx

The error occurs for the following reasons:

1. The username or password is incorrect.

2. The connection times out.

To resolve this issue, if the error message contains the following content, you can run the obd env set OBD_DISABLE_RSA_ALGORITHMS 1 command to disable the rsa-sha2-512 and rsa-sha2-256 algorithms.

Open ssh connection \Exception (client): Unable to agree on a pubkey algorithm for signing a 'ssh-rsa' key!

If the error message does not contain the preceding content, you can manually connect to the corresponding server by using the SSH command and verify whether the connection information is correct.

• If the connection fails, check whether the connection information is correct and whether the server is configured properly.

• If the connection succeeds, modify the connection information based on the following methods.

  • If you deploy the cluster in command-line mode, run the obd cluster edit-config <deploy_name> command to open the configuration file. Add or modify the user parameter in the configuration file. Save the file and run the command that is displayed in the command line to make the modification take effect.

  • If you deploy the cluster in GUI mode, click Previous. On the Node Configuration page (when you deploy OceanBase Database) or the MetaDB Configuration page (when you deploy OCP), modify the Deployment User Configuration section.

OBD-1015: Unable to confirm the primary-standby relationship, rerun with "--ignore-standby" option if you want to proceed despite the risks

The error occurs because the primary-standby relationship of the cluster or tenant cannot be confirmed during the execution of the command.

To resolve this issue, check whether the cluster or tenant has a standby tenant in another cluster based on the following methods.

• If the cluster is available, run the following command to check whether a standby tenant exists. Here is an example where the cluster name is test.

  obd cluster tenant show test -g
  

• If the cluster is unavailable, run the following command to check the primary-standby relationships of the cluster or tenant. Here is an example where the cluster name is test.

  cat ~/.obd/cluster/test/inner_config.yaml
  

  Based on the output, run the following command on the corresponding cluster to check whether the primary-standby relationship still exists. Here is an example where the cluster name is test-standby.

  obd cluster tenant show test-standby -g
  

Based on the detection results and the current operation, you can perform the following operations. For more information about the commands, see Cluster commands.

• If the current operation is an upgrade and the cluster or tenant has a standby tenant, you can first upgrade the standby tenant and then upgrade the primary tenant. If the cluster has both a primary and a standby tenant, you can run the obd cluster tenant switchover command to switch the primary tenant to the standby tenant. After the primary and standby tenants are upgraded, run the obd cluster tenant switchover command again to switch them back.

  Note

  If the current cluster is upgraded to the same version, you can directly run the command and add the --ignore-standby option to skip the check.

• If the current operation is not an upgrade (such as destroy, redeploy, or drop) and the cluster or tenant has a standby tenant, you can use the following methods to remove the primary-standby relationship.

  • Run the obd cluster tenant decouple command on the standby tenant. The standby tenant will be independent as the primary tenant.

  • Run the obd cluster stop command on the cluster of the primary tenant to stop the cluster, and then run the obd cluster tenant failover command on the standby tenant. The standby tenant will be independent as the primary tenant.

  • Run the obd cluster tenant drop command on the standby tenant. The standby tenant will be deleted.

• If the cluster or tenant has no standby tenant, or you can accept the risk that the standby tenant is unavailable, you can run the command again and add the --ignore-standby option to skip the check.

OBD-1016: (xx.xx.xx.xx) failed to get xxx using command "sysctl -a"

Cause: The connection is abnormal or the operating system does not support the sysctl -a command.

Solution: You can retry the operation.

OBD-1017: (xx.xx.xx.xx) The value of the "xxx" must be xxx

Cause: The kernel parameters of the operating system are not in the recommended range.

To ensure the stability of OceanBase Database in a production environment, obd checks the system environment and kernel parameters before starting OceanBase Database. This check ensures that the system configuration meets the recommended parameter settings of OceanBase Database. If the configuration does not meet the recommended standards, and if production_mode is set to true or the --strict-check option is enabled during command execution, the instance will be identified as a production environment. In this case, an error report will be triggered, and the startup process will be terminated. Conversely, if the configuration does not meet the recommended standards, and if production_mode is set to false or the --strict-check option is not enabled, only a warning will be issued, and the startup process will not be terminated.

Solution: Based on the different environments, the following two solutions are available.

• If the environment is a production environment, you can use the sysctl -w {kernel parameter name}="recommended value" command or the echo "kernel parameter name=recommended value" >> /etc/sysctl.conf; sysctl -p command to modify the parameter configuration to meet the requirements.

• If the environment is a test environment and you do not have permission to modify the kernel parameters, you can use the obd cluster edit-config {deployname} command to modify the configuration file and set the production_mode parameter to false to skip the system parameter blocking check.

OBD-1018: could not change xxx

Cause: The specified configuration file contains a configuration item that cannot be modified.

Solution: You can check whether the specified configuration file contains configuration items such as user, depends, unuse_lib_repository, and auto_create_tenant.

OBD-1019: component xxx already in cluster

Cause: The component to be added already exists in the cluster. obd does not support adding existing components to the cluster.

Solution: You can execute the obd cluster edit-config command to open the configuration file and check the component information in the cluster to confirm whether the component to be added already exists.

OBD-1020: Update config for component xxx failed

Cause: The component configuration update failed.

Solution: Check the configuration file based on the error code information output.

OBD-1021: Component xxx is not in cluster

Cause: The component to be deleted does not exist in the cluster.

Solution: You can execute the obd cluster edit-config command to open the configuration file and check the component information in the cluster to confirm whether the component to be deleted exists.

OBD-1022: Component xxx still depends by xxx, could not remove

Cause: The component to be deleted is depended by other components in the cluster. obd does not support deleting components that have dependencies.

Solution: You can execute the obd cluster edit-config command to open the configuration file and check the dependencies between cluster components (depends).

OBD-1023: Failed to merge config: xxx

Cause: The configuration file merge failed.

Solution: Check the configuration file based on the error code information output.

OBD-1024: The cluster will have no remaining components

Cause: The component to be deleted is the only component in the cluster.

Solution: If you want to delete all components in the cluster, you can execute the obd cluster destroy command to destroy the cluster.

OBD-1025: ({ip}) {component} {key} invalid

Cause: The password configured for the corresponding component does not meet the requirements. obd checks the component based on the password requirements of the component. The specific password requirements are as follows.

• The admin login password of the OCP console (admin_password) must meet the following requirements:

  • The length is between 8 and 32 characters.

  • At least three of the following four types of characters are included: digits (0~9), uppercase letters (A~Z), lowercase letters (a~z), and special characters (~!@#%^&*_-+=|(){}[]:;,.?/$`'"<>\)

• The HTTP service authentication password of OBAgent (http_basic_auth_password) must meet the following requirements:

  • The length is between 0 and +∞.

  • It can contain digits (0~9), uppercase letters (A~Z), lowercase letters (a~z), and special characters (~^*{}[]_-+)

Solution: When you perform command-line operations, you can execute the obd cluster edit-config command to open the configuration file, modify the corresponding configuration item under the corresponding component in the configuration file, and save the changes. Then, copy and execute the restart command output by the command line.

OBD-1026: Could not modify {key} when the cluster is in the working status(production_mode is True, not support this operation)

Cause: The modified configuration item takes effect only after you execute the obd cluster redeploy command. This command will destroy the cluster and redeploy it. The data in the cluster will be lost. Therefore, in a production environment (production_mode=true), you cannot modify configuration items that require redeployment to take effect.

Solution: By default, the production_mode parameter is set to true when you deploy an OceanBase cluster. If the current cluster is not in a production environment and you can accept the risk of data loss, you can execute the obd cluster edit-config command to set the production_mode parameter to false. Then, restart the cluster based on the output command and modify the configuration item again.

OBD-1027: If you are sure the directory can be emptied, run obd cluster deploy -f {deploy_name} to perform forced deployment

Cause: The directory specified in the configuration file is not empty.

Solution: You can choose to clear the directory or configure a new empty directory. The specific methods are as follows:

• Clear the directory: You can manually clear the directory (the error code OBD-1002 will output the corresponding information) or add the -f parameter after the deployment command to automatically clear the directory (copy the command output by the error code).

• Configure a new directory: You can execute the obd cluster edit-config <deploy name> command to open the configuration file and configure a new directory for the corresponding configuration item (the error code OBD-1002 will output the corresponding information).

OceanBase deployment errors

OBD-2000: x.x.x.x not enough memory

Cause: The available memory is insufficient.

Solution: The memory of obd is calculated based on the MemAvailable value. If there are cacheable resources, you can use the following commands to release them:

sudo sysctl -w vm.drop_caches=1
# or
sudo echo 1 > /proc/sys/vm/drop_caches

If the memory is still insufficient, execute the obd cluster edit-config <deploy name> command to open the configuration file and adjust the memory_limt and system_memory parameters. Generally, memory_limt/3 ≤ system_memory ≤ memory_limt/2.

OBD-2001: server can not migrate in

Cause: The number of available units is less than --unit-num.

Solution: Modify the value of --unit-num. You can run the following command to view the number of available units.

select count(*) num from oceanbase.__all_server where status = 'active' and start_service_time > 0

OBD-2002: failed to start x.x.x.x observer

Cause: This error can be caused by the following two common reasons:

• The value of memory_limit is less than 8 GB.

• The value of system_memory is too large or too small. Generally, memory_limt/3 ≤ system_memory ≤ memory_limt/2.

Solution:

• If the error is caused by the two reasons mentioned above, you can adjust the parameters based on the corresponding reasons.

OBD-2003: not enough disk space for clog. Use redo_dir to set other disk for clog, or reduce the value of datafile_size

Cause: The disk usage rate exceeds the specified threshold.

• If you use automatic deployment, the disk usage rate must not exceed 72%.

• If you use manual deployment, the disk usage rate must not exceed 64% without changing the configurations.

Solution: You can adjust the disk storage based on the deployment method. The following two methods are available:

• If you use command-line deployment, run the obd cluster edit-config <deploy name> command to open the configuration file, modify the disk-related parameters (such as datafile_size, memory_limit, and log_disk_size), save the changes, and then execute the command output in the command line to apply the changes.

• If you use the graphical interface for deployment, click Previous to go to the Cluster Configuration (when you deploy OceanBase Database) or MetaDB Configuration (when you deploy OCP) page. In the More Configurations section of the Cluster Configuration tab, modify the disk-related parameters (such as datafile_size, memory_limit, and log_disk_size).

OBD-2004: Invalid: xxx is not a single server configuration item

Cause: The modified parameter is a global parameter and cannot be modified for a specific server.

Solution: You can move the parameter to the global section.

OBD-2005: Failed to register cluster. xxx may have been registered in xxx

Cause: The cluster registration failed, or the cluster has already been registered.

Solution: First, check whether the appname parameter is configured. If the appname parameter is not configured, the cluster cannot be registered to obconfigserver. Then, based on whether the cluster is deployed, handle the following two scenarios:

• Scenario 1: If the cluster to be registered to obconfigserver is an undeployed OceanBase cluster, comment out the obconfig_url parameter, start the cluster, and then execute the obd cluster edit-config command to configure the obconfig_url parameter. Currently, it is not supported to register an undeployed cluster to obconfigserver.

• Scenario 2: If the cluster to be registered to obconfigserver is a running cluster, check whether the obconfig_url parameter is configured correctly.

  • If the obconfig_url parameter is not configured correctly, execute the obd cluster edit-config command to open the configuration file and configure the correct Config URL for the obconfig_url parameter.

  • If you confirm that the obconfig_url parameter is configured correctly and want to forcibly overwrite the existing registration, add the -f parameter when you execute the obd cluster start command to overwrite the registered cluster.

OBD-2006: x.x.x.x has more than one network interface. Please set devname for x.x.x.x

Cause: The server has multiple network interfaces, and obd cannot obtain the devname.

Solution: Based on the deployment method, use one of the following two solutions:

• If you use command-line deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file, add or modify the devname parameter in the configuration file, save the changes, and then execute the command output in the command line to apply the changes.

• If you use the graphical interface for deployment, click Previous to go to the Cluster Configuration (when you deploy OceanBase Database) or MetaDB Configuration (when you deploy OCP) page. In the More Configurations section of the Cluster Configuration tab, set the devname.

OBD-2007: x.x.x.x xxx fail to ping x.x.x.x

Cause: The nodes cannot ping each other.

Solution:

1. Check whether the network is accessible between the nodes.

2. Check whether the network configuration (devname/local_ip) matches the actual configuration. You can run the ip addr command to view the IP address and network interface card (NIC) mapping. If they do not match, you can modify them based on the deployment method.

   • For command-line deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Add or modify the devname parameter in the configuration file. Save the file and run the command output in the command line to make the modification take effect.

   • For GUI deployment, click Previous. On the Cluster Configuration page (when you deploy OceanBase Database) or the MetaDB Configuration page (when you deploy OCP), click More Settings in Cluster Configuration and set devname.

3. If the error is operation not permitted, check the ping file permissions. You can run the sudo chmod u+s /usr/bin/ping command to modify the ping file permissions.

4. If the error is No such file or directory, it indicates that the ping command is not available in your environment. You can run the sudo yum install iputils or sudo apt-get install iputils-ping command to install the ping command.

OBD-2008: Cluster clocks are out of sync

Cause: The clocks of the cluster nodes are out of sync.

Solution: Synchronize the clocks of the cluster nodes.

OBD-2009: x.x.x.x: when production_mode is True, xxx can not be less then xxx

Cause: When the production mode is enabled, the values of parameters such as __min_full_resource_pool_mem and memory_limit cannot be less than the specified values.

Solution:

• For non-production environments, run the obd cluster edit-config <deploy_name> command to open the configuration file. Set the production_mode parameter to False. Save the file and run the command output in the command line to make the modification take effect.

• For production environments, run the obd cluster edit-config <deploy_name> command to open the configuration file. Set the __min_full_resource_pool_mem and memory_limit parameters to values greater than the specified values. Save the file and run the command output in the command line to make the modification take effect.

OBD-2010: x.x.x.x: system_memory too large. system_memory must be less than memory_limit/memory_limit_percentage

Cause: The value of the system_memory parameter is too large. The value of this parameter must be less than the value of the memory_limit parameter or the product of the memory_limit_percentage parameter and the total_memory parameter.

Solution: You can use the following methods based on the deployment method.

• For command-line deployment, run the obd cluster edit-config <deploy name> command to open the configuration file. Set the system_memory parameter. Save the file and run the command output in the command line to make the modification take effect.

• For GUI deployment, click Previous. On the Cluster Configuration page (when you deploy OceanBase Database) or the MetaDB Configuration page (when you deploy OCP), click More Settings in Cluster Configuration and set system_memory.

OBD-2011: x.x.x.x: fail to get memory info.\nPlease configure 'memory_limit' manually in configuration file

Cause: The server cannot obtain the memory information.

Solution: You can use the following methods based on the deployment method.

• For command-line deployment, run the obd cluster edit-config <deploy_name> command to open the configuration file. Set the memory_limit parameter. Save the file and run the command output in the command line to make the modification take effect.

• For GUI deployment, click Previous. On the Cluster Configuration page (when you deploy OceanBase Database) or the MetaDB Configuration page (when you deploy OCP), click More Settings in Cluster Configuration and set memory_limit.

OBD-2014: xx.xx.xx.xx cpu does not support avx, Please change the server

Cause: The operating system of the OBServer node does not support the AVX instruction set.

Solution: Replace the server with one that supports the AVX instruction set.

OBD-2015: {server}: Failed to modify the configuration of the automatic startup

Cause: The current user does not have sudo privileges, so the automatic startup configuration cannot be modified. When you configure the automatic startup of an OBServer node (enable_auto_start), make sure that the user has sudo privileges.

Solution: You can configure sudo privileges for the corresponding user on each OBServer node. Use the root user or another user with sudo privileges to open the /etc/sudoers file:

[root@test001 ~]# vim /etc/sudoers

Add the following content at the end of the /etc/sudoers file. This example shows how to configure passwordless sudo privileges for the admin user.

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL
admin       ALL=(ALL)       NOPASSWD: ALL

OBD-2016: Primary tenant {primary_tenant} does not have complete logs, and cannot create a standby cluster

Cause: When creating a standby tenant based on the network, the primary tenant must have complete log information. You can log in as the root user of the primary tenant and execute the SELECT LS_ID, BEGIN_LSN FROM oceanbase.GV$OB_LOG_STAT WHERE ROLE = 'LEADER' ; command to view the information. If the value of BEGIN_LSN is 0, it indicates that the current log stream replica has complete logs since its creation.

Solution: If the primary tenant does not have complete logs since its creation, you can archive the logs of the primary tenant and create a standby tenant based on the archive. For more information, see Create a standby tenant based on an archive. After the standby tenant is created, if you still want to synchronize logs over the network, you can refer to Switch the log synchronization method between primary and standby tenants to change the synchronization method to network-based synchronization.

OBD-2017: Continuous log synchronization is not enabled

Cause: After creating a standby tenant based on an archive, you must enable log synchronization for the standby tenant to ensure that logs are synchronized between the primary and standby tenants.

Solution: You can execute the obd cluster tenant recover command to enable log synchronization for the standby tenant. For more information about this command, see Cluster commands. Search for obd cluster tenant recover to view the command details.

OBD-2018: For the standby tenant created by {primary_tenant} in log archive mode, it is prohibited to create a network-mode standby tenant for this standby tenant

Cause: In a cascading primary-standby scenario, after creating a standby tenant (B_a) for the primary tenant (A_a) based on an archive, you cannot create a standby tenant (C_a) for the standby tenant (B_a) based on the network.

Solution: If you want to create a standby tenant for the standby tenant (B_a) created based on an archive, you can choose to create it based on an archive. For more information, see Create a standby tenant based on an archive.

OBD-2020: Multiple observer nodes on the same server are not supported by the auto start feature

Cause: The auto start feature (enable_auto_start set to true) is enabled when multiple observer processes are started on the same server.

Solution: You can choose one of the following solutions based on your actual situation.

• Solution 1: Execute the obd cluster edit-config <deploy name> command to open the configuration file and set enable_auto_start to false to disable the auto start feature for observer processes.

• Solution 2: Start only one observer process on the server.

OBD-2021: Permission denied. Current user {user} on server {ip} requires sudo privileges

Cause: The configured user does not have sudo privileges.

Solution: You can choose one of the following solutions based on your actual situation.

• Solution 1: Configure sudo privileges for the user on the corresponding node.

• Solution 2: Click Previous Step and update the user configuration on the Node Configuration page using a user with sudo privileges.

  Note

  Make sure that the user exists on each node and has sudo privileges.

Errors related to testing

OBD-3000: parse cmd failed

Cause: The initialization file for mysqltest must be an SQL file ending with .sql.

Solution: Please check whether the --init-sql-files parameter meets this requirement.

OBD-3001: xxx.sql not found

Cause: The corresponding initialization file cannot be found during mysqltest initialization.

Solution: Please check whether the --init-sql-files file is included in the --init-sql-dir directory.

OBD-3002: Failed to load data

Cause: This error can occur for various reasons, with the following two being the most common:

1. The tenant has insufficient resources or is under heavy load.

2. An error occurs in the data construction script.

Solution:

For case 1, you can use a tenant with a larger resource specification or adjust the values of parameters such as warehouses and load-workers to reduce the construction pressure.

For case 2, since the data construction script is provided by TPC, you can try re-running the script.

OBD-3003: Failed to run TPC-C benchmark

Cause:

1. The test process is killed due to a timeout after being stuck.

2. The TPC-C test command returns an error.

Solution:

• You can directly rerun the test or reduce the test pressure by adjusting parameters such as terminals and then rerun the test.

• If you are not using the obtpcc package provided by the official website, please use obtpcc for the test.

OBAgent-related errors

OBD-4000: Fail to reload x.x.x.x

Cause: The http_basic_auth_password of the node does not match the http_basic_auth_password stored in obd, causing obd to be unable to access obagent.

Solution: If you confirm that the two are consistent, please check whether the modified options include configuration items not supported by the current version or whether the configuration item names are written incorrectly.

OBD-4001: Fail to send config file to x.x.x.x

Cause: There are two reasons for this error. Please check them in order.

• Whether the disk space of the obagent home_path is sufficient.

• Whether the user (the default user is the current user if not specified) in the configuration file has write permissions for the obagent home_path.

Solution: You can solve this issue in the following two ways.

• Run the obd cluster edit-config <deploy_name> command to add or modify the user information, save the changes, and execute the command output in the command line to make the changes take effect.

• Log in to the target machine and grant the current account write permissions for the corresponding directory.

OBD-4002: {servers}: Failed to obtain the configuration of the OceanBase database component

Cause: The server information (the servers section) configured for the OBAgent component in the configuration file does not match the OceanBase database component, causing OBAgent to be unable to obtain the OceanBase database configuration.

Solution: You can execute the obd cluster edit-config command to modify the servers section of the OBAgent or OceanBase database component in the configuration file to make them consistent.

ODP-related error codes

OBD-4100: x.x.x.x need config "rs_list" or "obproxy_config_server_url"

Cause: The server cannot obtain the rs_list or obproxy_config_server_url information.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file, add or modify the rs_list or obproxy_config_server_url parameter, save the changes, and then execute the command provided in the command line output to apply the changes.

OBD-4101: failed to start x.x.x.x obproxy: xxx

Cause: The ODP failed to start.

Solution: Further analysis is required based on the prompt.

ODP-4102: When the value of client_session_id_version is set to xx, the valid range of proxy_id is xxxx

Cause: The value of the proxy_id parameter is out of the valid range. The valid range of the proxy_id parameter depends on the value of the client_session_id_version parameter, as follows:

• If the value of the client_session_id_version parameter is set to 1, the valid range of the proxy_id parameter is [1,255].

• If the value of the client_session_id_version parameter is set to 2, the valid range of the proxy_id parameter is [1,8191].

Solution: Run the obd cluster edit-config command to open the configuration file, modify the value of the proxy_id parameter. After modifying and saving the configuration file, copy and execute the corresponding command provided in the command line output to restart the cluster.

Grafana-related errors

OBD-4200: x.x.x.x grafana admin password should not be 'admin'

Cause: The password for the admin user in the Grafana component should not be 'admin'.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file, add or modify the password information, save the changes, and then execute the command provided in the output to apply the changes.

OBD-4201: x.x.x.x grafana admin password length should not be less than 5

Cause: The password for the admin user in the Grafana component must be at least 5 characters long.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file, add or modify the password information, save the changes, and then execute the command provided in the output to apply the changes.

OCP-related errors

OBD-4350: The Server have running task

Cause: During the upgrade of OCP, there are running tasks in OCP.

Solution: To avoid task interruption during the upgrade of OCP, wait for the tasks to complete and then recheck.

OBD-4351: The Server have gone

Cause: During the upgrade of OCP, the host is not online.

Solution: Query the status of the host managed by OCP.

• If the host status is Submitted, the host is newly added. Please wait for the host addition task to complete and then recheck.

• If the host status is Offline, you can try to reinstall the OCP Agent.

OBD-4352: Metadb version not fewer than V2.2.50

Cause: The MetaDB version of OCP is lower than 2.2.50.

Solution: You can upgrade the MetaDB of OCP to the latest Long Term Support (LTS) version.

OBD-4353: {server}: Excessive deviation between machine time and ob time

Cause: The host time and MetaDB time are inconsistent.

Solution: You can follow the steps below to troubleshoot and resolve the issue.

1. Execute the following commands on the corresponding host to check whether the host has installed a time synchronization service (Chrony or NTP).

   rpm -qa | grep chrony   # Check whether the Chrony service is installed.
   rpm -qa | grep ntp      # Check whether the NTP service is installed.
   

   Based on the output, you can handle the issue in the following two ways.

   • If the output contains the corresponding version information, the corresponding time synchronization service is installed. Proceed to step 2.

   • If the output contains no information, the corresponding time synchronization service is not installed. If neither the Chrony nor the NTP service is installed, install a time synchronization service first. For more information about how to install and configure the Chrony or NTP service, see the examples shared on the Internet. Here is a brief description.

     • Execute the following command to install a time synchronization service. You can install either the Chrony or the NTP service.

       sudo yum install -y chrony    # Install the Chrony service.
       sudo yum install -y ntp       # Install the NTP service.
       

     • Execute the following command to start the time synchronization service.

       systemctl start chronyd     # Start the Chrony service.
       systemctl start ntpd        # Start the NTP service.
       

     • Recheck the precheck.

2. Execute the following commands to check whether the time synchronization process (chronyd or ntpd) has exited abnormally.

   systemctl status chronyd      # Check the status of the Chrony service.
   systemctl status ntpd         # Check the status of the NTP service.
   

   Based on the output, you can handle the issue in the following two ways.

   • If the output contains the information that the service is inactive (dead), the time synchronization service is abnormal. Try to execute the following commands to restart the service.

     systemctl restart chronyd      # Restart the Chrony service.
     systemctl restart ntpd         # Restart the NTP service.
     

     After the service is restarted, you can retry the deployment.

OBD-4354: {user}@{server}: Not exist

Cause: The OCP startup user does not exist.

Solution: Use another startup user or create a startup user on the host where OCP is deployed.

OBD-4355: {user}@{ip}: user xxx not in sudoers or sudoers file not exist

Cause: The user cannot execute the sudo command without a password.

Solution: Configure the user to execute the sudo command without a password or use another user with the sudo privilege.

OBD-4356: failed to connect meta db

Cause: The MetaDB cannot be connected.

Solution: Check whether the connection string of MetaDB is correct.

OBD-4357: database in jdbc_url is not exist

Cause: The database specified in the JDBC connection does not exist.

Solution: Create the corresponding database in MetaDB.

OBD-4358: unmatched jdbc url, skip meta db connection check

Cause: The JDBC URL format is incorrect.

Solution: Check the jdbc_url configuration and make sure that it meets the following format: "^jdbc:\S+://(\S+?)(|:\d+)/(\S+)".

OBD-4359: {server}: ocp-server need java with version xxx and update release must greater than 161

Cause: The Java version does not meet the requirements of OCP.

Solution: Upgrade the Java version to 1.8.0_161 or later.

OBD-4360: {server}: clockdiff not exists. Please install clockdiff manually

Cause: The clockdiff command does not exist on the host.

Solution: Install clockdiff.

OBD-4361: tenant(xxx) already exist

Cause: The tenant already exists.

Solution: You can log in to MetaDB and delete the tenant with the same name, or use a different tenant name.

OBD-4362: {server}:{path} access failed for current user, {server}:{cur_path} access succeed, please run chmod -R 755 {cur_path}

Cause: The user does not have the required permissions to operate the directory.

Solution: Execute the output chmod command to grant the user the required permissions for the directory.

OBD-4363: ocp server xxx needs to use xxx with version xxx or above

Cause: The corresponding component needs to be used in the deployment of OCP.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file, modify the version of the component that caused the error (the version parameter), save the changes, and then run the command output in the command line to make the changes take effect.

OBD-4364: (xx.xx.xx.xx) not enough memory

Cause: The server does not have enough memory.

Solution: The following solutions are available.

• If the server does not have enough memory, you can reduce the value of the memory_size parameter or replace the server with one that has sufficient memory. The following methods are available for reducing the value of the memory_size parameter, depending on the deployment method.

  • For command-line deployment, run the obd cluster edit-config <deploy name> command to open the configuration file, reduce the value of the memory_size parameter, save the changes, and then run the command output in the command line to make the changes take effect.

  • For GUI deployment, click Previous to go back to the MetaDB Configuration page. In the More Settings section under Cluster Configuration, reduce the value of the memory_size parameter.

• If the remaining memory resources on the server are insufficient, you can try to release cached memory by running the following commands:

  sudo sysctl -w vm.drop_caches=1
  # or
  sudo echo 1 > /proc/sys/vm/drop_caches
  

OBD-4365: x.x.x.x xxx not enough disk space. (Avail: xxx, Need: xxx)

Cause: The server does not have enough disk space.

Solution: Please check and clean up the disk space.

OBD-4366: There is not enough {resource}. (Avail: {avail}, need: {need})

Cause: The metadb resource is insufficient.

Solution: The following solutions are available.

• For command-line deployment, run the obd cluster edit-config <deploy name> command to open the configuration file, increase the values of the memory_limit, cpu_count, and log_disk_size parameters under the oceanbase-ce component, save the changes, and then run the command output in the command line to make the changes take effect.

• For GUI deployment, click Previous to go back to the MetaDB Configuration page. In the More Settings section under Cluster Configuration, increase the values of the memroy_limit, cpu_count, and log_disk_size parameters.

OBD-4367: The allocated memory for the provided meta database is currently insufficient for creating a tenant

Cause: The metadb memory resources are insufficient to create a tenant.

Solution: You can increase the metadb memory or reduce the tenant memory. The following solutions are available.

• For command-line deployment, run the obd cluster edit-config <deploy name> command to open the configuration file, increase the value of the memory_limit parameter under the oceanbase-ce component, or reduce the values of the memory_size parameters under the ocp_meta_tenant and ocp_monitor_tenant subcomponents of the oceanbase-ce component. Save the changes and then run the command output in the command line to make the changes take effect.

• For GUI deployment, click Previous to go back to the MetaDB Configuration page. In the More Settings section under Cluster Configuration, increase the value of the memroy_limit parameter; or, on the OCP Configuration page, reduce the memory values of the metadata tenant and monitoring data tenant configuration modules.

OBD-4368: The allocated memory for the provided meta database is currently insufficient for creating a tenant

Cause: The metadb memory resources are insufficient to create a tenant.

Solution: You need to reduce the tenant memory. The following solutions are available.

• For command-line deployment, run the obd cluster edit-config <deploy name> command to open the configuration file, reduce the values of the memory_size parameters under the ocp_meta_tenant and ocp_monitor_tenant subcomponents of the ocp-server-ce component. Save the changes and then run the command output in the command line to make the changes take effect.

• For GUI deployment, click Previous Step and reduce the memory values of the metadata tenant and monitoring data tenant configuration modules on the OCP Configuration page.

obconfigserver-related errors

OBD-4401: Failed to start x.x.x.x ob-configserver

Cause:

1. Cause 1: An internal error occurred during the startup of obconfigserver, and the service terminated.

2. Cause 2: The listening port for obconfigserver is not open on the target deployment server, making it inaccessible.

Solution: Log in to the target deployment server and execute the following command to determine the cause of the error.

ps -ef | grep $home_path/bin/ob-configserver 

$home_path is the working directory of obconfigserver. If no running obconfigserver process is found in the output, the cause is Cause 1. Otherwise, the cause is Cause 2.

For Cause 1, you can check the error message keywords in the $home_path/log/ob-configserver.log file. Most of the time, the error is due to an incorrect connection_url configuration when using the sqlite3 database type. Correct the corresponding error configuration.

For Cause 2, you can use the following two solutions.

• If you are using a cloud server, log in to the corresponding cloud server and add the server's IP address to the allowlist.

• If you are using a self-built server, enable port listening based on the operating system version.

OBD-4402: x.x.x.x ob-configserver config error

Cause: An error was detected in the obconfigserver-related configuration.

Solution: Based on the specific description, check whether the configuration items are missing or the parameters are non-compliant. The following scenarios may occur:

• When using a VIP, are both vip_address and vip_port set?

• Are both database_type and connection_url configured? (Note: In the case of using the sqlite3 database type, connection_url is optional.)

• Is the database_type configured correctly? The database_type parameter only supports the values mysql or sqlite3.

• When using the sqlite3 database type, is connection_url configured as an absolute path?

OBD-4403: ob-configserver connect to sqlite failed: x.x.x.x: /xxx/xxx/xxx: permission denied

Cause: When using sqlite3 as the database, the user specified in the configuration file (defaulting to the current user if not specified) does not have write permissions for the directory specified in connection_url.

Solution: You can use the following two methods to resolve this issue.

• Run the obd cluster edit-config <deploy_name> command to open the configuration file, add or modify the user information, save the changes, and then execute the command outputted in the command line to make the changes take effect.

• Log in to the target machine and grant write permissions for the corresponding directory to the current account.

OBD-4404: ob-configserver connect to mysql failed: xxx: failed url to connect to database: xxx

Cause: When database_type is set to mysql, the database specified in connection_url cannot be connected.

Solution: Verify whether the database specified in connection_url can be connected. If not, replace it with a database that can be connected.

OBD-4405: When you configure multiple ob-configserver servers, please set vip_address and vip_port

Cause: When deploying obconfigserver on multiple server nodes, VIP configuration is not used.

Solution: When deploying obconfigserver on multiple nodes, it is recommended to deploy a load balancer in advance and configure vip_address and vip_port. Depending on the deployment method, follow the following two methods to configure the relevant parameters.

• For command-line deployment, execute the obd cluster edit-config <deploy name> command to open the configuration file and set vip_address and vip_port in the configuration file. Save the changes and execute the command outputted in the command line to make the configuration take effect.

• For GUI deployment, click Previous to go to the Cluster Configuration page, open the More Settings section in the Component Configuration module, and configure vip_address and vip_port.

oblogproxy related error messages

OBD-4501: oblogproxy xxx needs to use xxx with version xxx or above

Cause: The oblogproxy component requires a specific version of the corresponding component.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file, modify the version of the corresponding component (version), save the changes, and then execute the command provided in the output to apply the changes.

Binlog service-related errors

OBD-4601: OBBinlog {obbinlog_version} needs to use {comp} with version {min_version} or above

The error occurs because the version of OceanBase Database in the meta database is too low. When OceanBase Database is used as the meta database, the version must be V4.2.1.0 or later.

To resolve this error, run the obd cluster edit-config <deploy name> command to open the configuration file, and configure the version information under the oceanbase-ce component. The format is as follows:

oceanbase-ce:
  version: 4.3.3.0

After the configuration is saved, run the printed obd cluster redeploy <deploy name> command to redeploy the Binlog service.

OBD-4602: Deploy <deploy_name> need depends ob-configserver

The error occurs because the obconfigserver component is not deployed in the current business cluster.

To resolve this error, you can use one of the following methods:

• Method 1: Specify the OBServer nodes of the cluster by using the --rs option of the obd binlog create command. You can run the SHOW PARAMETERS LIKE 'rootservice_list'; command to obtain the value of the OBServer nodes. Then, create a Binlog instance for the specified tenant.

  Note

  The --rs option of the obd binlog create command was added in obd V4.2.0. This option is supported only when obbinlog is V4.3.5 or later.

• Method 2: Run the obd cluster component add <deploy_name> -c <ob-configserver.config> command to add the obconfigserver component. For more information, see the Add a component section in Expand and change components.

  If ODP is deployed separately, you can use the root@proxysys user to connect to ODP and run the alter proxyconfig set obproxy_config_server_url='10.10.10.1:8080/services?Action=GetObProxyConfig'; command after the component is added. Then, restart ODP.

  Note

  In the preceding example, 10.10.10.1:8080 specifies the IP address and port of obconfigserver. You must replace them with the actual IP address and port.

OBD-4603: The Binlog service for the community version can only be used with the community version of the OceanBase database

The error occurs because the Binlog service for the community version can be used only with OceanBase Database of the community version.

To resolve this error, deploy the Binlog service for the enterprise version if you want to use the Binlog service for OceanBase Database of the enterprise version. For more information, see Deploy obbinlog.

OBD-4604: The Binlog service for the enterprise version can only be used with the enterprise version of the OceanBase database

The error occurs because the Binlog service for the enterprise version can be used only with OceanBase Database of the enterprise version.

To resolve this error, deploy the Binlog service for the community version if you want to use the Binlog service for OceanBase Database of the community version. For more information, see Deploy obbinlog.

OMS error messages

OBD-4701: failed to connect meta db

Error reason: When you deploy OMS by using an external database as the MetaDB, the system fails to connect to the MetaDB.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file and check whether the MetaDB configuration is correct. If not, update it with the correct information.

OBD-4702: failed to connect influxdb

Error reason: The system fails to connect to the corresponding time-series database based on the access information of the configured time-series database.

Solution: Run the obd cluster edit-config <deploy_name> command to open the configuration file and check whether the time-series database configuration is correct. If not, update it with the correct information.

OBD-4703: ({ip}) {disk} not enough disk space

Error reason: When you upgrade OMS in online mode, you must specify the path for storing the upgrade files. The disk space of the specified path is insufficient.

Solution: Configure a directory with sufficient disk space.

OBD-4704: HA is enabled, please disable it before upgrade

Error reason: Before you upgrade OMS, you must disable the high availability (HA) module. The HA module ensures the stability and continuity of the data transmission link.

Solution: Disable the HA module before you upgrade OMS. Procedure:

1. Log in to the OMS Community Edition console.

2. In the left-side navigation pane, choose System Management > System Parameters.

3. On the System Parameters page, find ha.config.

4. Click the edit icon next to the Value field of the parameter.

5. In the Modify Value dialog box, set enable to false to disable the HA module.

SQL-related error codes

OBD-5000: SQL execute failed

Cause: The SQL statement failed to execute.

Solution: Determine the solution based on the specific circumstances.

obdiag-related error messages

OBD-6000: Failed to execute obdiag command. You may not have obdiag installed.

Cause: The obdiag component is not installed.

Solution: You can install the obdiag component by running the following command:

obd tool install obdiag
source ~/oceanbase-diagnostic-tool/init.sh

OBD-6001: obdiag must contain dependent components xxxx

Cause: The obdiag component is not installed. The obdiag service on obd is used to manage OceanBase or ODP clusters deployed through obd. If no OceanBase or ODP cluster is deployed on obd, this error will be reported.

Solution: Install the dependent components of obdiag, which means at least one OceanBase or ODP cluster must be registered on obd. You can run the obd cluster list command to view all clusters registered on obd.

OBD-6002: obdiag options xxx format error, please check the value : xxx

Cause: The value of the obdiag command parameter does not meet the requirements.

Solution: You can use the -h option after the corresponding command to view the parameter requirements of the obdiag command and pass in the correct obdiag parameter format, as shown in the example.

# example
obd obdiag gather -h

obd obdiag gather log -h

OBD-6003: Failed to execute obdiag function xxx

Cause: The obdiag command execution failed, which is likely due to executing a non-existent obdiag command.

Solution: You can run the following command to view the obdiag commands supported by obd.

obd obdiag -h