External table common issue troubleshooting guide

Background information

After the JNI framework is introduced in OBServer as the bridge for connecting to external data lakes, various environment initialization issues may arise.

Issue 1: Error when creating an HDFS or ODPS JNI external table

Error 1: ERROR 11056: JNI env could not be found

Cause

The JNI environment initialization may fail due to the following reasons:

1. ob_java_home is incorrectly set, preventing the proper loading of the libjvm.so dynamic library. This library is crucial for initializing the JVM environment.

2. ob_java_opts is incorrectly set. Due to the limitations in the integration between the CPP framework and the JNI framework, it is necessary to forcibly set specific parameters to start the JVM process. For more information, see ob_java_opts.

3. The libhdfs.so library file (the communication bridge between OceanBase Database and JVM) is placed in a path that does not match the _ob_additional_lib_path. This needs to be set accurately.

Solution

1. Manually verify the current java home settings, focusing on whether the current user has access permissions for this path. It is possible that java home exists, but the OBServer startup user does not have access permissions to this path.

2. Check if any required configuration items are missing in the ob_java_opts parameter. For more information, see ob_java_opts.

3. Verify if the _ob_additional_lib_path is correctly set and if the OBServer startup user has access permissions to this path.

Error 2: ERROR 11032: HDFS: path not found

Cause

The HDFS: path not found error typically occurs in the following scenarios:

1. In a Kerberos-authenticated HDFS environment, if the user does not have sufficient permissions to access a specific HDFS external table, an -11032 error may occur.

2. If Kerberos authentication is correctly configured, but the HDFS: path not found error still occurs, it may be due to an incorrect path specified in the CREATE EXTERNAL TABLE statement, pointing to a non-existent path.

3. Initially, the external table access is normal, but after some time, an 11032 error may occur if there are bad blocks in the HDFS path corresponding to the external table. This happens when the HDFS DataNode reports that the data block is faulty and inaccessible, triggering the error.

Solution

1. Confirm that the Kerberos authentication keytab and principal are valid and that the corresponding user has access permissions to the HDFS cluster. Reconfigure the external table permissions and set up Kerberos authentication.

2. Verify that the specified HDFS path is correct and exists. Re-establish the HDFS path mapping for the external table.

3. Report to the customer's operations team to confirm the bad block situation. You can choose to adjust the bad block or replace the environment for further verification.

Error 3: ERROR 4016: Internal Error

Cause

The Internal Error may occur due to the following reasons:

1. When querying an external table, an Internal Error may occur if the actual data types of the external table do not match the expected types. For example, if a String type column is mapped to an Int type, some field data conversion may fail.
2. Other unexpected execution paths may trigger an Internal Error.

Solution

1. Check the metadata type mapping in the CREATE EXTERNAL TABLE statement and the corresponding external table data files.

2. Review the relevant logs, capture the error log stack, and confirm the specific details of the error.