Command-line options

OBLOADER allows you to specify the information required for import in command-line options. For more information about the options and their scenarios and examples, see Options and Scenarios and examples.

Overview

Option styles

OBLOADER supports the Unix and GNU styles of command-line options.

• Unix style: An option is prefixed with a hyphen and each option is a single character, such as ps -e. In this style, you can omit the space between an option and its value, such as -p******.

• GNU style: An option is prefixed with double hyphens and each option is a single character or a string, such as ps --version. An option and its value must be separated with a space, such as --table 'test'.

Option categories

Command-line options in OBLOADER are classified into basic options and advanced options.

• Basic options: general options of OBLOADER, including connection options (such as the database connection method), feature options (such as the file format, database object type, and storage path), and other options.

• Advanced options: feature options (such as the timestamp format, allowlist- and blocklist-based table/column filtering settings, and error handling method), performance options, and other options.

Required options

When you use OBLOADER to import data, you must specify at least the connection options, format option, database object type option, and storage path option.

Here is a sample statement:

$./obloader -h xx.x.x.x -P 2883 -u test@mysql#cluster_a -p ****** -D USERA --csv --table '*' -f /output

In this example, -h, -P, -u, -p, and -D are connection options, the --csv option specifies the file format, the --table option specifies the database object type, and the -f option specifies the storage path.

Basic options

Connection options

OBLOADER can read data from and write data to an OceanBase database only after it is connected to the database. You can connect to an OceanBase database by specifying the following options:

Import type

Bypass import

OBLOADER V4.2.6 and later support bypass import.

• --rpc-port= rpc_port_num

  The RPC port for connecting to the OBServer node. This option is used in combination with the --direct and --parallel options to specify to connect to the RPC port of an OBServer node to import data in bypass import mode.

  Note

  • This option applies only to OceanBase Database V4.2.0 RC2 and later.
  • You can query the RPC port of an OBServer node in the following ways:
    • In the case of connection to the ODP server:
      • In ApsaraDB for OceanBase, the RPC port of the ODP server is 3307 by default.
      • In OceanBase Database, the RPC port of the ODP server is 2885 by default. You can change the port by using the -s option when you start ODP.
    • In the case of direct connection to an OBServer node, you can query the DBA_OB_SERVER system view in the sys tenant for the RPC port of the OBServer node, which is 2882 by default.

• --direct

  Specifies to use the bypass import mode. This option is used in combination with the --rpc-port and --parallel options.

  Note

  • The bypass import mode of OBLOADER does not support binary data types.
  • The bypass import mode of OBLOADER supports direct connections to an OBServer node and connections through OceanBase Database Proxy (ODP). The version requirements are as follows:
    • OBServer node: V4.2.0 or later.
    • ODP: V4.1.3, or V4.3.0 or later while the OBServer node version must be V4.2.1 or later.

• --parallel= parallel_num

  The degree of parallelism (DOP) for loading data in bypass import mode. This option is used in combination with the--rpc-port and --direct options.

Feature options

File formats

Decompress data for import

OBLOADER V4.3.0 and later allow you to decompress CSV, CUT, POS, and SQL files exported by OBDUMPER before import. You can complete related configurations by using the following command-line options:

• --compress

  Indicates whether the to-be-imported files have been compressed by OBDUMPER during export. It is a Boolean value.

• --compression-algo

  Indicates the compression algorithm by which the to-be-imported files are compressed by OBDUMPER during export. The value is a string. Valid values are zstd, zlib, gzip, and snappy. The default value is zstd.

• --compression-level

  The compression level for the corresponding compression algorithm. The value is an integer. Compression levels supported for different compression algorithms are as follows:

  • zstd: 1 to 22

  • zlib and gzip: -1 to 9.

  • snappy: You cannot specify a compression level for this compression algorithm.

Database object types

• --all

  Imports all supported database object definitions and table data. When this option is used in combination with --ddl, all database object definitions are imported. When this option is used in combination with --csv, --sql, --cut, or --pos, all data files in the specified format are imported. To import all database object definitions and table data, you can specify the --all and --ddl options with a data format option.

  Note

  The --all option is mutually exclusive with any database object options. It cannot be specified together with other database object options. If both the --all option and a database object option are specified, the --all option will be executed first.

• --table-group 'table_group_name [,table_group_name...]' | --table-group '*'

  Imports table group definitions. This option is similar to the --table option, except that this option does not support data import.

• --table 'table_name [,table_name...]' | --table '*'

  Imports table definitions or table data. When this option is used in combination with the --ddl option, only table definitions are imported. When this option is used in combination with any data format option, only table data is imported. To specify multiple tables, separate the table names with commas (,). By default, for OceanBase Database in Oracle mode, the table names are in uppercase, and for OceanBase Database in MySQL mode, the table names are in lowercase. For example, for OceanBase Database in Oracle mode, both --table 'test' and --table 'TEST' indicate the table named TEST. For OceanBase Database in MySQL mode, both --table 'test' and --table 'TEST' indicate the table named test. If table names are case-sensitive, enclose them in brackets ([ ]). For example, --table '[test]' indicates the table named test, while --table '[TEST]' indicates the table named TEST. If the table name is specified as an asterisk (*), all table definitions or table data is imported.

  Notice

  When you use a control file to import data, the table name specified in the --table option must be in the same letter case as that in the database. Otherwise, the control file fails to take effect.

• --view 'view_name [, view_name...]' | --view '*'

  Imports view definitions. This option is similar to the --table option, except that this option does not support data import.

• --trigger 'trigger_name [, trigger_name...]' | --trigger '*'

  Imports trigger definitions. This option is similar to the --table option, except that this option does not support data import.

• --sequence 'sequence_name [, sequence_name...]' | --sequence '*'

  Imports sequence definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.

• --synonym 'synonym_name [, synonym_name...]' | --synonym '*'

  Imports synonym definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.

• --type 'type_name [, type_name...]' | --type '*'

  Imports type definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.

• --type-body 'typebody_name [, typebody_name...]' | --type-body '*'

  Imports type body definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.

• --package 'package_name [, package_name...]' | --package '*'

  Imports package definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.

• --package-body 'packagebody_name [, packagebody_name...]' | --package-body '*'

  Imports package body definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.

• --function 'function_name [, function_name...]' | --function '*'

  Imports function definitions. This option is similar to the --table option, except that this option does not support data import.

• --procedure 'procedure_name [, procedure_name...]' | --procedure '*'

  Imports stored procedure definitions. This option is similar to the --table option, except that this option does not support data import.

Storage path

• -f 'file_path', --file-path= 'file_path'

  The absolute path for storing data files. When data is stored in another file system instead of the local file system, you can use the --tmp-path option to specify a temporary file directory.

  OBLOADER allows you to import database object definitions and table data from local disks, Alibaba Cloud Object Storage Service (OSS), Amazon Simple Storage Service (S3), Apache Hadoop, Tencent Cloud Object Storage (COS), and Huawei Cloud Object Storage Service (OBS). The syntax is as follows:

  [scheme://host]path[?parameters]
  parameters: key[=value],...
  

  Component	Description
  scheme	The storage scheme. Alibaba Cloud OSS, Amazon S3, Apache Hadoop, Tencent Cloud COS, and Huawei Cloud OBS are supported. If the scheme is not Alibaba OSS, Amazon S3, Hadoop, Tencent Cloud COS, or Huawei Cloud OBS, an error will be returned.
  host	The name of the storage space. When you import data from Alibaba Cloud OSS, Amazon S3, Tencent Cloud COS, or Huawei Cloud OBS, the host parameter specifies a bucket. For more information, see OSS Bucket. When you import data from Apache Hadoop, the host parameter specifies a Hadoop node, which is in the <ip>:<port> or <cluster_name> format.
  path	The data storage path in the storage space. The path must start with a slash (/).
  parameters	The parameters required for the request. The value can be a single key or multiple key-value pairs.

  The following table describes the supported parameters.

  Parameter	Value required?	Description	Supported storage scheme	Supported version
  endpoint	Yes	The endpoint of the region where the OSS host resides. The endpoint for accessing Amazon S3, Tencent Cloud COS, or Huawei Cloud OBS.	OSS/S3/COS/OBS	OSS: V4.2.0 S3: V4.2.5 COS/OBS: V4.3.0
  region	Yes	The physical location of the bucket.	S3/COS/OBS	S3: V4.2.0 COS/OBS: V4.3.0
  storage-class	Yes	The storage class of Amazon S3.	S3	V4.2.0
  access-key	Yes	The AccessKey ID used to access the bucket.	OSS/S3/COS/OBS	OSS/S3: V4.2.0 COS/OBS: V4.3.0
  secret-key	Yes	The AccessKey secret used to access the bucket.	OSS/S3/COS/OBS	OSS/S3: V4.2.0 COS/OBS: V4.3.0
  hdfs-site-file	Yes	A hdfsSiteFile configuration file. The configuration file contains the configuration information of Apache Hadoop, such as the block size and number of replicas. Storage and access rules are set for Apache Hadoop based on the configuration information.	Apache Hadoop	V4.2.1
  core-site-file	Yes	A hdfsSiteFile configuration file. The configuration file contains the core configuration information of the Hadoop cluster, such as the URI of the file system and the default file system of Apache Hadoop.	Apache Hadoop	V4.2.1
  principal	Yes	The identifier for identity authentication in Kerberos.	Apache Hadoop	V4.2.1
  keytab-file	Yes	The absolute path of the Keytab file, which authorizes users or services to access system resources.	Apache Hadoop	V4.2.1
  krb5-conf-file	Yes	The path where the Kerberos configuration file resides.	Apache Hadoop	V4.2.1

  Here are some examples:

  • Import data from the local disk

    -f '/home/admin/foo/bar'
    

  • Import data from Amazon S3

    -f 's3://bucket/path?region={region}&access-key={accessKey}&secret-key={secretKey}'
    

  • Import data from Alibaba Cloud OSS

    -f 'oss://mybucket/foo/bar?endpoint=myendpoint&access-key=myak&secret-key=mysk'
    

  • Import data from Apache Hadoop

    -f 'hdfs://***.*.*.*:9000/chang/parquet?hdfs-site-file=/data/0/zeyang/hdfs-site.xml&core-site-file=/data/0/zeyang/core-site.xml
    

  • Import data from Tencent Cloud COS

    -f 'cos://mybucket/foo?region=ap-shanghai&access-key=myak&secret-key=mysk'
    -f 'cos://mybucket/foo?endpoint=yourendpoint&access-key=myak&secret-key=mysk'
    

  • Import data from Huawei Cloud OBS

    -f 'obs://mybucket/foo?region=cn-north-1&access-key=myak&secret-key=mysk'
    -f 'obs://mybucket/foo?endpoint=yourendpoint&access-key=myak&secret-key=mysk'
    

• --ctl-path 'control_path'

  The absolute path on a local disk for storing control files. You can configure built-in preprocessing functions in a control file. Data will be preprocessed by these functions before being imported. For example, the functions can perform case conversion and check the data for empty values. For the use of control files, see Data processing. When you specify this option on the command line, enclose the value in single quotation marks (' '), Example: --ctl-path '/home/controls/'.

• --log-path 'log_path'

  The output directory for the operational logs of OBLOADER. If this option is not specified, OBLOADER operational logs are stored in the directory specified by the -f option. Redirection is not required for log output, unless otherwise specified.

Other options

• -H, --help

  Shows the help information about the tool.

• -V, --version

  Shows the version of the tool.

Advanced options

Feature options

Timestamp formats

• --nls-date-format 'date-format-string'

  The format of dates in database connections in Oracle mode of OceanBase Database. Default value: YYYY-MM-DD HH24:MI:SS.

• --nls-timestamp-format 'timestamp-format-string'

  The format of timestamps in database connections in Oracle mode of OceanBase Database. Default value: YYYY-MM-DD HH24:MI:SS:FF9.

• --nls-timestamp-tz-format 'timestamp-tz-format-string'

  The format of timestamps that contains a time zone in database connections in Oracle mode of OceanBase Database. Default value: YYYY-MM-DD HH24:MI:SS:FF9 TZR.

Allowlist- and blocklist-based filtering

• Allowlist- and blocklist-based table filtering

  • --exclude-table 'table_name [, table_name...]'

    Excludes the specified tables from the import of table definitions or table data. Fuzzy match on table names is supported,

    such as --exclude-table 'test1,test*,*test,te*st'

    The preceding example specifies to exclude the following tables from the import of table definitions or table data:

    • test1

    • All tables with a table name starting with test

    • All tables with a table name ending with test

    • All tables with the table name starting with te and ending with st

  • --exclude-data-types 'datatype [, datatype...]'

    Excludes the specified data types from the import of table data.

  • --file-regular-expression

    The regular expression that is used to match the file name during an import. This option applies only to single-table import. For example, --file-regular-expression ".*" specifies to use the ".*" regular expression to match all files in the directory. In the expression, * matches zero or more characters. --file-regular-expression ".*\\.csv" specifies to use the ".*\.csv" regular expression to match files whose file names end with ".csv". In the expression, "\." matches the period ".".

• Allowlist- and blocklist-based column filtering

  • --exclude-column-names 'column_name [, column_name...]'

    Excludes the specified columns during data import.

    Notice

    • The letter case of the specified column name must be the same as that of the column name in the table structure.
    • In the imported data file, the excluded columns must have no corresponding data, and the imported columns must be in the same order as the columns in the table.

Error handling

• --max-discards int_num

  The maximum amount of duplicate data allowed in a single table to import. If duplicate data in a table exceeds the specified maximum, data import for this table stops. The import failure on the table is logged. Data import for other tables is not affected. The default value is -1, indicating that data import does not stop for duplicate data.

  Note

  This option takes effect only when the table contains a primary key or a unique key and duplicate data exists in the table.

• --retry

  Resumes the import task from the breakpoint. We recommend that you specify this option to resume an import task that has more than 80% of data imported, so that you do not have to start a new import. Duplicate data errors may occur during the resumed import. If only a small part of the data has been imported, you can clear tables and start a new import, which is more efficient.

  Notice

  • The `load.ckpt` file is a savepoint file generated when the tool runs and is located in the directory specified by the -f option. You cannot use this option if the `load.ckpt` file does not exist.
  • This option is inapplicable to bypass import.

• --max-errors int_num

  The maximum number of errors allowed for an import. If the number of import errors of a table exceeds the specified maximum, data import for this table stops. The import failure on the table is logged. This option can be set to 0, -1, or any positive integer N. If you set this option to -1, the errors are ignored and the import continues. The default value is 1000.

• --strict= 'strict_string'

  Specifies to control the impact of dirty data on the process end status during import. The default value is true, which indicates that the program is to end in the failed state (System exit 1) when the imported data contains a Bad Record or Discard Record error. The value false indicates that the final end state (System exit 0) of the program is not affected when the imported data contains a Bad Record or Discard Record error.

  Note

  This option can be used in combination with the --max-discards or --max-errors option, which specifies to skip duplicate data or errors and proceed with the process when the amount of duplicate data or the number of errors is within the specified range. For more information, see Error handling.

• --replace-data

  Replaces duplicate data in the table. This option is only applicable to tables that have primary keys or unique keys with the NOT NULL constraint. If a file has a large amount of duplicate data, such as more than 30% of the total data volume, we recommend that you clear the table and import it again. The performance of data replacement is lower than that of import after table clearing. This option can be used only in combination with the --cut, --csv, and --sql options, and does not take effect for the --ddl option.

  Notice

  • When the file and the table have duplicate data, the data in the table is replaced with the data in the file.
  • For tables without primary keys or unique keys, this option appends data to the tables.
  • If you do not need to replace duplicate data, do not specify this option. Otherwise, business may be affected.

Performance options

• --rw float_num

  The proportion of data file parsing threads. The default value is 1. You can use this option in combination with the --thread option to calculate the number of file parsing threads by using the following formula: Number of file parsing threads = Value of --thread × Value of --rw.

• --slow float_num

  The threshold for triggering a slow import. When the memory usage of OceanBase Database reaches 75%, OBLOADER slows down to prevent an excessively high memory usage on the database. Default value: 0.75.

• --pause float_num

  The threshold for triggering an import pause. When the memory usage of OceanBase Database reaches 85%, OBLOADER pauses data import to prevent issues caused by an excessively high memory usage on the database. Default value: 0.85.

• --batch int_num

  The number of records for a batch of transactions. We recommend that you set this option to a value inversely proportional to the table width. Do not set this option to an excessively high value, which may cause database memory overflow. Default value: 200.

  Note

  • In OBLOADER V4.2.0 and later, the default value of the --batch option can be adapted based on the Java virtual machine (JVM) memory.

  • In OBLOADER V4.2.8.1 and V4.2.8.2, the number of parameters in an executed prepared statement may exceed the upper limit, causing a core dump in ODP. In this case, you can manually set this option to an appropriate value.

• --thread int_num

  The number of concurrent threads allowed. This option corresponds to the number of import threads. You can use this option with the --rw option to calculate the number of file parsing threads based on the following formula: Number of file parsing threads = Value of --thread × Value of --rw. The default value is the value of the number of CPU cores multiplied by 2. If the number of CPU cores is greater than 16, the maximum value is 32. OceanBase Database executes DDL statements in sequence. Therefore, you do not need to specify this option when importing database object definitions.

• --block-size int_num

  The size of a file to be imported. When specifying this option, you do not need to explicitly specify the unit. The default unit is MB. By default, OBLOADER automatically splits a large file into multiple logical subfiles (or blocks) sized 64 MB. The logical subfiles do not occupy additional storage space. Default value: 64.

• --max-tps int_num

  The maximum import speed. You can specify this option to ensure import stability.

• --max-wait-timeout int_num

  The timeout period of waiting for a database major compaction to complete. When specifying this option, you do not need to explicitly specify the unit. The default unit is hour. When the database is under a major compaction, OBLOADER stops data import and waits up to the specified period. The default value is 3.

Other options

• --truncate-table

  Truncates tables in the destination database. This option can be used only in combination with a data format option. When being used in combination with the --all or --table '*' option, this option specifies to truncate all tables in the destination database. If you want to truncate only some of the tables, you can explicitly specify them in the format of --table 'test1,test2,[....]'. When being used in combination with the --with-data-files option, this option specifies to truncate tables that have the corresponding data files.

  Notice

  • If the --all or --table '*' option is used in combination with the --truncate-table option, the tool truncates all tables in the destination database, even if no corresponding data files of the tables exist in the directory specified by the -f option.
  • Do not use this option to truncate the destination database or destination tables. We recommend that you manually truncate tables to avoid impacts on business.

• --with-data-files

  When being used in combination with the --truncate-table or --delete-from-table option, this option specifies to truncate or clear tables that have corresponding data files before data import. If not used in combination with these options, this option does not take effect.

• --delete-from-table

  Clears tables in the destination database before the import. This option can be used only in combination with a data format option. When being used in combination with the --all or --table '*' option, this option specifies to clear all tables in the destination database. If you want to clear only some of the tables, you can explicitly specify them in the format of --table 'test1,test2,[....]'. When being used in combination with the --with-data-files option, this option specifies to clear tables that have corresponding data files only.

  Notice

  • If the --all or --table '*' option is used in combination with the --delete-from-table option, the tool clears all tables in the destination database, even if no corresponding data files of the tables exist in the directory specified by the -f option.
  • Do not use this option to clear the destination database or destination tables, especially tables with large data volumes. We strongly recommend that you delete table data manually based on your business requirements to avoid impacts on your business.

• --replace-object

  Replaces existing database object definitions during the import. For tables and synonyms, existing definitions are deleted and then new ones are created. For functions and stored procedures, existing definitions are replaced by using the CREATE OR REPLACE statement. This option can be used only in combination with the --ddl or --mix option, and does not take effect for --csv, --sql, and other data format options.

  Notice

  • If an object already exists in the destination database, the object definition is forcibly replaced with the object definition stored in the import file.
  • If you do not need to replace objects in the destination database, do not use this option. Otherwise, business may be affected.

Options