Deploy the OceanBase Database Java SDK

OceanBase Database now supports connecting to a file system as an external data source using the Java SDK. This feature uses the JNI framework, so you need to deploy the Java SDK environment for OceanBase Database.

Environment configuration

All environment configuration and dependency installation operations must be performed by the admin user. Please switch to the admin user by using the following command:

su - admin

Deploy and configure the Java environment

1. Download the OpenJDK installation package from Download address.

   Notice

   Please use the latest version of OpenJDK 11.

   This example uses the OpenJDK 11.0.29+7 version. The specific operation steps are as follows:

   [admin@xxx /home/admin/rpm]# wget https://builds.openlogic.com/downloadJDK/openlogic-openjdk/11.0.29+7/openlogic-openjdk-11.0.29+7-linux-x64.tar.gz
   

2. Decompress the installation package normally.

   Here is an example:

   [admin@xxx /home/admin/rpm]# sudo tar -zxvf openlogic-openjdk-11.0.29+7-linux-x64.tar.gz
   

   The installation path is as follows:

   /home/admin/rpm/openlogic-openjdk-11.0.29+7-linux-x64
   

   Notice

   This path is used to configure the Java home directory on the node where the OBServer runs, which is the value of the ob_java_home parameter.

Install dependencies

To use the HDFS/ODPS external table feature of OceanBase Database, you must also install the following components:

• devdeps-hdfs-sdk RPM package:

  This package contains dynamic link library files required for the runtime of HDFS/ODPS external tables. It provides core interfaces for interaction between the JVM environment and JNI (Java Native Interface). This component serves as a communication bridge between the Java Virtual Machine and the local external table Java SDK, ensuring the stable operation of the external table feature.

• devdeps-java-extensions RPM package:

  This package integrates core dependency libraries (JAR files) required for HDFS external tables and other external data sources, such as ODPS. The extension package contains a complete Java runtime dependency chain, ensuring compatibility and performance optimization of the external table feature in distributed scenarios.

Deploy and configure the HDFS.so dynamic library

1. Obtain the devdeps-hdfs-sdk RPM installation package.

   • For Enterprise Edition users, contact Technical Support to obtain the devdeps-hdfs-sdk RPM installation package.
   • For Community Edition users, click the development-kit/ directory on the OceanBase Database image page to enter the development tool resource directory and download the devdeps-hdfs-sdk RPM installation package.

2. After obtaining the installation package, install it by using the following command:

   sudo rpm -Uvh devdeps-hdfs-sdk-3.3.6-xxxxx.xxx.xxx.rpm
   

   Here is an example:

   sudo rpm -Uvh devdeps-hdfs-sdk-3.3.6-112024123116.el7.x86_64.rpm
   

3. Check whether the installation meets the expected results.

   The libhdfs.so and libhdfs.so.0.0.0 files must exist, and the corresponding soft links must be normal.

   Here is an example:

   $ll /usr/local/oceanbase/deps/devel/lib
   total 376
   lrwxrwxrwx 1 root root     16 Dec 24 19:49 libhdfs.so -> libhdfs.so.0.0.0
   -rwxr-xr-x 1 root root 384632 Dec 24 19:09 libhdfs.so.0.0.0
   

Deploy and configure the jar package path

1. Obtain the devdeps-java-extensions RPM installation package.

   • For Enterprise Edition users, contact Technical Support to obtain the devdeps-java-extensions RPM installation package.

   • For Community Edition users, click the development-kit/ directory on the OceanBase Database image page to enter the development tool resource directory and download the devdeps-java-extensions RPM installation package.

     Notice

     • For OceanBase Database V4.3.5 BP1 and earlier: Download the devdeps-java-extensions RPM installation package of version 1.0.0.
     • For OceanBase Database V4.3.5 BP2 and later: Download the devdeps-java-extensions RPM installation package of version 1.0.1.
     • For OceanBase Database V4.4.0: Download the devdeps-java-extensions RPM installation package of version 1.0.1.

2. After obtaining the installation package, install it by using the following command:

   sudo rpm -Uvh devdeps-java-extensions-x.x.x-xxxxxxxxxxxx.xxx.xxxxxx.rpm --prefix=/user_install_directory
   

   Here is an example:

   sudo rpm -Uvh devdeps-java-extensions-1.0.0-122025032514.el7.x86_64.rpm --prefix=/home/admin/oceanbase
   

3. Check whether the installation meets the expected results.

   Check whether the oceanbase-odps-connector-jar-with-dependencies.jar file exists in the /home/admin/oceanbase/jni_packages/v1.0.0 directory.

   Here is an example:

   $ll /home/admin/oceanbase/jni_packages/v1.0.0
   total 52756
   drwxr-sr-x 4 root root     4096 Dec 24 20:25 hadoop
   drwxr-xr-x 3 root root     4096 Dec 24 20:25 lib
   -rw-r--r-- 1 root root 54008720 Dec 24 19:52 oceanbase-odps-connector-jar-with-dependencies.jar
   

   Notice

   This path is used to configure the path of the executable dependency JAR package that can be loaded by the JVM when the OBServer starts, which is the value of the ob_java_connector_path parameter.

(Optional) Restart the observer process

When using the HDFS/ODPS external table feature, you need to configure the corresponding OBServer server. The configuration steps are as follows:

Step 1: Set the configuration items related to the Java environment

1. Enable the Java environment for accessing the SDK (Java SDK) of external tables.

   Here is an example:

   ALTER SYSTEM SET ob_enable_java_env = true;
   

   For more information, see ob_enable_java_env.

2. Set the Java home directory on the OBServer node where the current OBServer is running.

   Note

   The path is obtained from the installation path of OpenJDK.

   Here is an example:

   ALTER SYSTEM SET ob_java_home = "/home/admin/rpm/openlogic-openjdk-11.0.29+7-linux-x64";
   

   For more information, see ob_java_home.

3. Set the path of the executable dependency JAR packages that can be loaded by the JVM when the OBServer starts.

   Note

   The path is obtained from the RPM installation path of the JAR package.

   Here is an example:

   ALTER SYSTEM SET ob_java_connector_path = "/home/admin/oceanbase/jni_packages/v1.0.0";
   

   For more information, see ob_java_connector_path.

4. Set the configuration items related to the Java environment startup.

   1. Create the corresponding log folder path.

      mkdir -p /home/user/jvmlogs
      mkdir -p /home/user/jvmlogs/heapdumps
      

   2. Set the JVM startup configuration items for Java runtime.

      Here is an example:

      ALTER SYSTEM SET ob_java_opts="-Xmx2048m -Xms2048m -XX:-CriticalJNINatives -Djdk.lang.processReaperUseDefaultStackSize=true -Xrs -XX:+HeapDumpOnOutOfMemoryError -Xloggc:/home/admin/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/admin/heapdumps/ -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+TieredCompilation -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses ";
      

      For more information, see ob_java_opts.

      Notice

      • Changing this configuration requires restarting the observer process. Since the current HDFS integration uses direct memory copying from the HDFS data stream to the C++ memory heap, you can reduce the -Xmx2048m -Xms2048m setting.
      • GC log files are generated only when the configuration folder path exists; if the path does not exist, no GC log files will be generated.

5. (Optional) Set the configuration items for using the Java SDK for ODPS.

   Notice

   The HDFS Java SDK does not require this configuration item, and other configuration items are the same.

   If you want to use the Java SDK, set _use_odps_jni_connector to true.

   ALTER SYSTEM SET _use_odps_jni_connector = true;
   

Step 2 (Optional): Decompress the hdfs-sdk package without sudo permissions

1. If you cannot obtain sudo permissions to execute the installation command, you can manually decompress the devdeps-hdfs-sdk RPM installation package using the rpm2cpio command. You can place the required packages in the desired path as needed.

   Here is an example:

   rpm2cpio devdeps-hdfs-sdk-3.3.6-xxxxx.xxx.xxx.rpm | cpio -idmv
   

2. Specify the target path.

   After decompressing the installation package, you can use the mv command to move the decompressed files to a custom path that is accessible to you (e.g., ~/hdfs_lib).

   Here is an example:

   mv ./usr/local/oceanbase/deps/devel/lib/* ~/hdfs_lib/
   

3. Obtain and confirm the absolute path.

   You can use the realpath command to view the absolute path of the custom path.

   Here is an example:

   realpath ~/hdfs_lib
   

   The result is as follows:

   /home/${user_name}/hdfs_lib
   

4. Configure the path variable for OceanBase Database.

   Here is an example:

   Log in to the sys tenant and execute the following command to register the custom path to the system configuration.

   Notice

   • When executing the following statement, replace ${user_name} in the example with the actual path.
   • The path must be an absolute path, and multiple paths must be separated by colons : without spaces.

   ALTER SYSTEM SET _ob_additional_lib_path = '/home/${user_name}/hdfs_lib';
   

Step 3: Restart the observer process

Stop the observer process on all servers and then restart the observer process.

1. Switch to the admin user.

   [admin@xxx /home/admin/oceanbase/etc]# su - admin
   

2. Stop the observer process.

   -bash-4.2$ kill -9 `pidof observer`
   

3. Start the observer process.

   -bash-4.2$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer
   

   Note

   You do not need to specify the startup parameters when you restart the observer process because the previous startup parameters have been written to the parameter file.

References

For more information about how to create an ODPS external table, see Create an external table (MySQL mode) or Create an external table (Oracle mode).