External Table Troubleshooting Guide

Background

After OBServer introduced the JNI framework as the bridge for connecting to external data lakes, you may encounter a number of issues related to environment initialization.

Issue 1: An error is reported when creating an HDFS/ODPS JNI external table

Error 1: ERROR 11056: JNI env could not be found

Cause

The JNI environment may fail to initialize for the following reasons:

1. ob_java_home is set incorrectly, which prevents the libjvm.so dynamic library from loading properly. This library is essential for initializing the environment and starting the JVM.

2. ob_java_opts is set incorrectly. Because there are certain limitations in the integration between the CPP framework and the JNI framework, the relevant property parameters must be explicitly set to start the JVM process, so as to avoid various configuration issues. For parameter settings, see ob_java_opts.

3. The libhdfs.so library file that the JNI environment depends on (the communication bridge between OceanBase Database and the JVM) is placed in a path that does not match _ob_additional_lib_path. It must be set correctly.

Solution

1. Manually verify whether the current java home setting exists, and pay particular attention to whether the current user has access permissions to that path. It is possible that java home exists but the user who starts OBServer does not have access permissions to that path.

2. Check against ob_java_opts whether any required items in ob_java_opts are missing.

3. Check whether the path set in _ob_additional_lib_path exists and whether the user who starts OBServer has access permissions to that path.

Error 2: ERROR 11032: HDFS: path not found

Cause

The HDFS: path not found error usually occurs in the following scenarios:

1. In an HDFS environment with Kerberos authentication enabled, when accessing an HDFS external table, the -11032 error occurs if insufficient permissions prevent access.

2. If Kerberos authentication is configured correctly but HDFS: path not found still occurs, the external table path specified in the CREATE statement may be incorrect and point to a nonexistent path.

3. If the external table can be accessed normally at first but the 11032 error occurs after running queries for some time, a file under the HDFS path corresponding to the external table may have a bad block. That is, the HDFS DataNode reports that the data block is corrupted and cannot be accessed, which triggers the error.

Solution

1. Confirm that the Kerberos keytab and principal are valid and that the corresponding user has access permissions to the HDFS cluster. Reconfigure the external table permissions and set up Kerberos authentication.

2. Confirm that the corresponding HDFS path is correct and exists, and re-establish the HDFS path mapping for the external table.

3. Report the issue to the customer's O&M team. After confirming the bad block, you can choose to fix the bad block or switch to another environment to continue verification.

Error 3: ERROR 4016: Internal Error

Cause

An Internal Error may occur for the following reasons:

1. An Internal Error occurs when running an external table query. This may be because the actual type set for the external table does not match the expected type. For example, mapping a String column to the Int type causes data conversion to fail for some fields.
2. Other unexpected program execution paths trigger the Internal Error.

Solution

1. Check whether the external table CREATE statement and the metadata type mapping of the corresponding external table data file are correct.

2. Check the related logs, capture the error log stack, and review and confirm the specific details that triggered the error.