External table troubleshooting guide

Background

After the OBServer introduced the JNI framework as a bridge for connecting to external data lakes, various environment initialization-related issues are inevitable.

Issue 1: Error when creating HDFS/ODPS JNI external tables

Error 1: ERROR 11056: JNI env could not be found

Cause

The following reasons may cause the JNI Env environment initialization to fail:

1. ob_java_home is set incorrectly, causing the libjvm.so dynamic library to fail to load properly. This dynamic library is an important library for initializing the JVM during environment initialization.

2. ob_java_opts is set incorrectly. Due to certain limitations in the connection between the CPP framework and the JNI framework, to avoid various usage setting issues, relevant attribute parameters must be forcibly set to start the JVM process. For parameter settings, see ob_java_opts.

3. The libhdfs.so library file (the communication bridge between OceanBase Database and the JVM) required by the JNI Env environment is placed in a path that does not match _ob_additional_lib_path. The setting must be accurate.

Solution

1. Manually confirm whether the current java home setting exists, and focus on verifying whether the current user has access permission to that path. The java home may exist, but the OBServer startup user may not have access permission to the path.

2. Check ob_java_opts to see if any required configuration items are missing from ob_java_opts.

3. Check whether the path set by _ob_additional_lib_path exists and whether the OBServer startup user has access permission to that path.

Error 2: ERROR 11032: HDFS: path not found

Cause

The HDFS: path not found error is typically seen in the following scenarios:

1. In an HDFS environment with Kerberos authentication enabled, if the permission is insufficient to access an HDFS external table, the -11032 error occurs.

2. If the Kerberos authentication configuration is correct but HDFS: path not found still occurs, the external table path specified in the external table creation statement may be incorrect and point to a non-existent path.

3. The external table access works fine initially, but the 11032 error occurs after running queries for some time. This may be because a file under the corresponding external table path in HDFS has bad blocks, meaning the HDFS DataNode reports that the data block is corrupted and cannot be accessed, triggering the error.

Solution

1. Confirm that the Kerberos authentication keytab and principal are valid and that the corresponding user has access permission to the HDFS cluster. Reconfigure the external table permissions and set Kerberos authentication.

2. Confirm that the corresponding HDFS path is correct and exists. Re-establish the HDFS path mapping for the external table.

3. Report to the customer's operations team. After confirming the data bad block situation, you can choose to adjust the bad blocks or replace the environment for further verification.

Error 3: ERROR 4016: Internal Error

Cause

The following reasons may cause Internal Error:

1. When executing an external table query, Internal Error occurs. This may be because the actual type of the external table does not match the expected type, for example, mapping a String type column to an Int type, causing some field data conversion to fail.
2. Other unexpected program execution paths triggered Internal Error.

Solution

1. Check whether the external table creation statement and the metadata type mapping of the corresponding external table data file are correct.

2. Check the relevant logs, capture the error log stack, and view and confirm the specific details of the error trigger.