Deploy the OceanBase Database Java SDK environment

OceanBase Database now supports connecting to a file system as an external data source by using the Java SDK. This feature uses the Java Native Interface (JNI) framework, so you need to deploy the Java SDK environment for OceanBase Database.

Environment configuration

All environment configuration and dependency installation operations must be performed as the admin user. Please switch to the admin user by using the following command:

su - admin

Configure the Java environment

1. Download the OpenJDK installation package from the download page.

   Notice

   Use the latest version of OpenJDK 11.

   This example uses OpenJDK 11.0.29+7. The specific operation steps are as follows:

   [admin@xxx /home/admin/rpm]# wget https://builds.openlogic.com/downloadJDK/openlogic-openjdk/11.0.29+7/openlogic-openjdk-11.0.29+7-linux-x64.tar.gz
   

2. Decompress the installation package.

   Example:

   [admin@xxx /home/admin/rpm]# sudo tar -zxvf openlogic-openjdk-11.0.29+7-linux-x64.tar.gz
   

   The installation path is as follows:

   /home/admin/rpm/openlogic-openjdk-11.0.29+7-linux-x64
   

   Notice

   This path is used to configure the Java home directory on the node where the OBServer runs, that is, the value of the ob_java_home parameter.

Install dependencies

To use the HDFS/ODPS external table feature of OceanBase Database, you also need to install the following components:

• devdeps-hdfs-sdk RPM package:

  This package contains dynamic link library files required for the runtime of HDFS/ODPS external tables. It provides core interfaces for interaction between the Java Virtual Machine (JVM) and the Java Native Interface (JNI). This component acts as a communication bridge between the JVM and the local external table Java SDK, ensuring the stable operation of the external table feature.

• devdeps-java-extensions RPM package:

  This package integrates core dependency libraries (JAR files) required for HDFS external tables and other external data sources such as ODPS. The extension package contains a complete Java runtime dependency chain, ensuring compatibility and performance optimization of the external table feature in distributed scenarios.

Deploy and configure the required HDFS.so dynamic library

1. Obtain the devdeps-hdfs-sdk RPM installation package.

   • For Enterprise Edition users, contact the technical support to obtain the devdeps-hdfs-sdk RPM installation package.

2. After obtaining the installation package, install it by using the following command:

   sudo rpm -Uvh devdeps-hdfs-sdk-3.3.6-xxxxx.xxx.xxx.rpm
   

   Example:

   sudo rpm -Uvh devdeps-hdfs-sdk-3.3.6-112024123116.el7.x86_64.rpm
   

3. Check whether the installation meets the expected results.

   The libhdfs.so and libhdfs.so.0.0.0 files must exist, and the corresponding symbolic links must be normal.

   Example:

   $ll /usr/local/oceanbase/deps/devel/lib
   total 376
   lrwxrwxrwx 1 root root     16 Dec 24 19:49 libhdfs.so -> libhdfs.so.0.0.0
   -rwxr-xr-x 1 root root 384632 Dec 24 19:09 libhdfs.so.0.0.0
   

Deploy and configure the dependency jar package path

1. Obtain the devdeps-java-extensions RPM installation package.

   • For Enterprise Edition users, contact Technical Support to obtain the devdeps-java-extensions RPM installation package.

   Notice

   • For OceanBase Database V4.3.5 BP1 and earlier: Download the devdeps-java-extensions RPM installation package of V1.0.0.
   • For OceanBase Database V4.3.5 BP2 and later: Download the devdeps-java-extensions RPM installation package of V1.0.1.
   • For OceanBase Database V4.4.0: Download the devdeps-java-extensions RPM installation package of V1.0.1.
   • For OceanBase Database V4.4.1: Download the devdeps-java-extensions RPM installation package of V1.0.2.

2. After obtaining the installation package, install it by using the following command:

   sudo rpm -Uvh devdeps-java-extensions-x.x.x-xxxxxxxxxxxx.xxx.xxxxxx.rpm --prefix=/user_install_directory
   

   Example:

   sudo rpm -Uvh devdeps-java-extensions-1.0.0-122025032514.el7.x86_64.rpm --prefix=/home/admin/oceanbase
   

3. Check whether the installation meets the expected results.

   Check whether the oceanbase-odps-connector-jar-with-dependencies.jar file exists in the /home/admin/oceanbase/jni_packages/v1.0.0 directory.

   Example:

   $ll /home/admin/oceanbase/jni_packages/v1.0.0
   total 52756
   drwxr-sr-x 4 root root     4096 Dec 24 20:25 hadoop
   drwxr-xr-x 3 root root     4096 Dec 24 20:25 lib
   -rw-r--r-- 1 root root 54008720 Dec 24 19:52 oceanbase-odps-connector-jar-with-dependencies.jar
   

   Notice

   This path is used to configure the path of the executable dependency JAR package that can be loaded by the JVM when the OBServer starts, that is, the value of the ob_java_connector_path parameter.

(Optional) Restart the observer process

When using the HDFS/ODPS external table feature, you need to configure the corresponding OBServer server. The configuration steps are as follows:

Step 1: Set the parameters related to the Java environment

1. Enable the Java environment for accessing the SDK (Java SDK) of external tables.

   Here is an example:

   ALTER SYSTEM SET ob_enable_java_env = true;
   

   For more information, see ob_enable_java_env.

2. Set the Java home directory on the OBServer node where the current OBServer is running.

   Note

   The path is obtained from the OpenJDK Java installation path.

   Here is an example:

   ALTER SYSTEM SET ob_java_home = "/home/admin/rpm/openlogic-openjdk-11.0.29+7-linux-x64";
   

   For more information, see ob_java_home.

3. Set the path to the executable dependency JAR packages that can be loaded by the JVM when the OBServer is started.

   Note

   The path is obtained from the JAR package RPM installation path.

   Here is an example:

   ALTER SYSTEM SET ob_java_connector_path = "/home/admin/oceanbase/jni_packages/v1.0.0";
   

   For more information, see ob_java_connector_path.

4. Set the parameters related to the Java environment startup.

   1. Create the corresponding log folder path.

      mkdir -p /home/user/jvmlogs
      mkdir -p /home/user/jvmlogs/heapdumps
      

   2. Set the JVM startup configuration for Java runtime.

      Here is an example:

      ALTER SYSTEM SET ob_java_opts="-Xmx2048m -Xmn2048m -XX:-CriticalJNINatives -Djdk.lang.processReaperUseDefaultStackSize=true -Xrs -XX:+HeapDumpOnOutOfMemoryError -Xloggc:/home/admin/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/heapdumps/ -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+TieredCompilation -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses ";
      

      For more information, see ob_java_opts.

      Notice

      • After you change this parameter, you must restart the observer process. The current HDFS integration uses memory copy, which directly copies HDFS data streams to the C++ memory heap. You can reduce the -Xmx2048m -Xms2048m setting.
      • GC log files are generated only when the configuration folder path exists. If the path does not exist, no GC log files are generated.

5. (Optional) Set the parameter for using the Java SDK of ODPS.

   Notice

   The HDFS Java SDK does not require this parameter. Other parameters are the same.

   If you want to use the Java SDK, set _use_odps_jni_connector to true.

   ALTER SYSTEM SET _use_odps_jni_connector = true;
   

Step 2 (Optional): Decompress the hdfs-sdk package without sudo privileges

1. If you cannot obtain sudo privileges to execute the installation command, you can manually decompress the devdeps-hdfs-sdk RPM installation package by using the rpm2cpio command. You can place the package in the required path as needed.

   Here is an example:

   rpm2cpio devdeps-hdfs-sdk-3.3.6-xxxxx.xxx.xxx.rpm | cpio -idmv
   

2. Specify the target path.

   After decompressing the installation package, you can use the mv command to move the decompressed files to a custom path accessible by users (for example, ~/hdfs_lib).

   Here is an example:

   mv ./usr/local/oceanbase/deps/devel/lib/* ~/hdfs_lib/
   

3. Obtain and verify the absolute path.

   You can use the realpath command to view the absolute path of the custom path.

   Here is an example:

   realpath ~/hdfs_lib
   

   The returned result is as follows:

   /home/${user_name}/hdfs_lib
   

4. Configure the OceanBase Database path variable.

   Here is an example:

   Log in to the sys tenant and execute the following command to register the custom path to the system configuration.

   Notice

   • When you execute the following statement, replace ${user_name} in the example with the actual path.
   • The path must be an absolute path. Multiple paths are separated by colons : and cannot contain spaces.

   ALTER SYSTEM SET _ob_additional_lib_path = '/home/${user_name}/hdfs_lib';
   

Step 3: Restart the observer process

Stop the observer process on all servers and restart it.

1. Switch to the admin user.

   [admin@xxx /home/admin/oceanbase/etc]# su - admin
   

2. Stop the observer process.

   -bash-4.2$ kill -9 `pidof observer`
   

3. Restart the observer process.

   -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer
   

   Note

   No startup parameters are required when you restart the observer process because the previous startup parameters have been written to the parameter file.

References

For detailed information about how to create an ODPS external table, see Create an external table (MySQL-compatible mode) or Create an external table (Oracle-compatible mode).