Command-line options

OBLOADER allows you to specify the information required for import in command-line options. For more information about the options and their scenarios and examples, see Options and Scenarios and examples.

Options

Connection options

OBLOADER can read data from and write data to an OceanBase database only after connecting to the database. You can connect to an OceanBase database by specifying the following options:

• -h host_name , --host= host_name

  The host IP address for connecting to ODP or a physical OceanBase Database node.
  
  

• -P port_num , --port= port_num

  The host port for connecting to ODP or a physical OceanBase Database node.
  
  

• -c cluster_name , --cluster= cluster_name

  The OceanBase cluster to connect to. If this option is not specified, a physical node of the database is connected. Relevant options, such as -h and -P, specify the IP address and port of the physical node. If this option is specified, ODP is connected. Relevant options, such as -h and -P, specify the IP address and port of ODP.
  
  

• -t tenant_name , --tenant= tenant_name

  The OceanBase Database tenant to connect to. For more information about tenants, see the official OceanBase documentation.
  
  

• -u user_name , --user= user_name

  The username for connecting to OceanBase Database. If you specify an incorrect username, OBLOADER cannot connect to the database.
  
  

• -p ' password' , --password=' password'

  The user password for connecting to OceanBase Database. If no password is set for the current database account, you do not need to specify this option. To specify this option on the command line, you must enclose the value in single quotation marks (' '), for example, -p'******'.

  Note

  If you are using Windows OS, enclose the value in double quotation marks (" "). This rule also applies to string values of other options.

• --rpc-port= rpc_port_num

  The RPC port for connecting to the OBServer node. This option is used with the --direct option, which specifies to connect to the RPC port of the OBServer node to import data in bypass import mode. You can query the RPC port of an OBServer node from the system table DBA_OB_SERVERS in the sys tenant.

  main id="notice" type='explain'>

  Note

  This option applies to only OceanBase Database V4.2.0 RC2 and later.

• --direct

  Specifies to use the bypass import mode. This option is used with the --rpc-port option.

  Note

  • OBLOADER V4.2.5 does not support bypass import for binary data types.
  • This option applies to only OceanBase Database V4.2.0 RC2 and later.

• --sys-user sys_username

  The username of a user with required privileges in the sys tenant, for example, root or proxyro. OBLOADER requires a special user in the sys tenant to query the metadata in system tables. The default value is root. You do not need to specify this option for OceanBase Database V4.0.0 and later.
  
  

• --sys-password ' sys_password'

  The password of a user with required privileges in the sys tenant. This option is used with the --sys-user option. By default, the password of the root user in the sys tenant is left empty. When you specify this option on the command line, enclose the value in single quotation marks (' '), for example, --sys-password '******'. You do not need to specify this option for OceanBase Database V4.0.0 and later.

  Note

  If this option is not specified, OBLOADER cannot query metadata in system tables, and the import features and performance may be greatly affected.

• --public-cloud

  Imports database objects or table data from an OceanBase cluster deployed in the public cloud. If you specify this option on the command line, you do not need to specify the -t or -c connection option. OBLOADER turns on the --no-sys option by default. For more information about the --no-sys option, see the corresponding option description. The use of the --public-cloud or --no-sys option will affect the import features, performance, and stability. OceanBase Database V2.2.30 and later versions support throttling on the server. Therefore, before you use the --public-cloud or --no-sys option, to ensure the stability of data import, you can run the following commands to set throttling thresholds as required on the server:

  alter system set freeze_trigger_percentage=50;
  alter system set minor_merge_concurrence=64;
  alter system set writing_throttling_trigger_percentage=80 tenant='xxx';
  

• --no-sys

  Imports database objects or table data from an OceanBase cluster deployed in the private cloud when you cannot provide the password of the sys tenant. Unlike the --public-cloud option, when you use the --no-sys option, you need to specify the -t connection option on the command line and also need to add the -c option to connect to the ODP service. In OceanBase Database V4.0.0 and earlier, if you do not specify the --public-cloud or --no-sys option, the --sys-user and --sys-password options must be specified for OBLOADER.
  
  

• --logical-database

  Specifies to import data by using ODP (Sharding). If you specify the --logical-database option on the command line, the definition of a random physical database shard is exported and the shard cannot be directly imported into the database. You need to manually convert the exported physical shard to a logical one before you import it to the database for business use.

Feature options

• -f ' file_path ' , --file-path= ' file_path '

  The absolute path on a local disk for storing data files. When you import data files from Alibaba Cloud Object Storage Service (OSS), you must specify the -f option to save the logs and binary files generated.
  
  

• --file-suffix ' suffix_name '

  The file name extension of data files to be imported. Generally, the file name extension is correlated with the file format. For example, a CSV file is usually named as xxx.csv. If you do not strictly follow the naming conventions, you may name a CSV file with any extension, such as .txt. In this case, OBLOADER cannot identify the file as a CSV file. This option is optional. A default value is available for each data format. The default file name extension is .csv for a CSV file, .sql for an SQL file, and .dat for a CUT or POS file. When you specify this option on the command line, enclose the value in single quotation marks (''), for example, --file-suffix '.txt'.
  
  

• --file-encoding ' encode_name '

  The file character set used when OBLOADER reads data files, which is not the database character set. When you specify this option on the command line, enclose the value in single quotation marks (' '), for example, --file-encoding 'GBK'. The default value is UTF-8.
  
  

• --ctl-path ' control_path '

  The absolute path on a local disk for storing control files. You can configure built-in preprocessing functions in a control file. Data will be preprocessed by these functions before being imported. For example, the functions can perform case conversion and check the data for empty values. For the use of control files, see Data processing. When you specify this option on the command line, enclose the value in single quotation marks (' '), for example, --ctl-path '/home/controls/'.
  
  

• --log-path ' log_path '

  The output directory for the operational logs of OBLOADER. If this option is not specified, OBLOADER operational logs are stored in the directory specified by the -f option. Redirection is not required for log output, unless otherwise specified.
  
  

• --ddl

  Imports DDL files. A DDL file stores the database object definitions. The file is named in the format of object name-schema.sql. When this option is specified, only database object definitions are imported, and table data is not imported.

  Notice

  Avoid comments or statements to enable/disable a feature in the file. If database objects depend on each other, the import may fail and manual intervention is required.

• --sql

  Imports data files in the SQL format. In an SQL file, data is stored in the format of INSERT statements. The file is named in the format of table name.sql. Each line of table data corresponds to an executable INSERT statement in an SQL file. An SQL file is different from a DDL file in terms of content format. We recommend that you use this option with the --table option. When this option is used with the --all option, OBLOADER imports only table data but not database object definitions.

  Notice

  The data cannot contain SQL functions, special characters, line breaks, and so on. Otherwise, the file may not be correctly parsed.

• --orc

  Imports data files in the ORC format. An ORC file stores data in the column-oriented format. The file is named in the format of table name.orc. For more information about ORC format definitions, see ORC Specification v1.
  
  

• --par

  Imports data files in the Parquet format. A Parquet file stores data in the column-oriented format. The file is named in the format of table name.parquet. For more information about Parquet format definitions, see File Format of Apache Parquet.
  
  

• --mix

  Imports MIX files. A MIX file stores a mix of DDL and DML statements, and does not have strict naming conventions.

  Notice

  MIX files do not have a strict format and feature a complex processing process and poor performance. Therefore, we recommend that you do not use MIX files.

• --csv

  Imports data files in the CSV format. In a CSV file, data is stored in the standard CSV format. The file is named in the format of table name.csv. For CSV format specifications, see the definitions in RFC 4180. Delimiter errors are the most common errors that occur in CSV files. Single or double quotation marks are usually used as the delimiter. If data in the file contains the delimiter, you must specify escape characters. Otherwise, OBLOADER fails to parse the data due to its incorrect format. We strongly recommend that you use the CSV format. We recommend that you use this option with the --table option. When this option is used with the --all option, OBLOADER imports only table data but not database object definitions.
  
  

• --pos

  Imports data files in the POS format. A POS file stores data with a fixed length in bytes. The file is named in the format of table name.dat. Data stored in each column of a POS file occupies a fixed number of bytes. A column value shorter than specified is padded with spaces. A column value longer than specified is truncated, in which case data may be garbled. We recommend that you use this option with the --table option. When this option is used with the --all option, OBLOADER imports only table data but not database object definitions. This is different from a format that stores data with a fixed length in characters.
  
  

• --cut

  Imports data files in the CUT format. A CUT file uses a character or a character string as the separator string. A CUT file is named in the format of table name.dat. How to distinguish the CUT format from the CSV format? A file in the CSV format uses a single character, which is usually a comma (,), to separate fields. A file in the CUT format usually uses a string, such as |@|, to separate fields. A file in the CSV format uses single quotation marks or double quotation marks as delimiters between fields, while a file in the CUT format does not have delimiters. We recommend that you use this option with the --table option. When this option is used with the --all option, OBLOADER imports only table data but not database object definitions.

  Notice

  In a CUT file, each data record is stored in an entire line. When a single character is used as the field separator, avoid special characters in the data, such as delimiters, carriage returns, and line breaks. Otherwise, OBLOADER cannot correctly parse the data.
  When you specify the --cut option on the OBLOADER command line, do not use the --trail-delimiter option if no field separator or separator string exists at the end of the data line in the file. Otherwise, a serious error occurs on OBLOADER.

• --table-group '*table_group_name [,table_group_name...]*'|--table-group '*'

  Imports table group definitions. This option is similar to the --table option, except that this option does not support data import.
  
  

• --all

  Imports all supported database object definitions and table data. When this option is used with the --ddl option, all database object definition files are imported. When this option is used with the --csv, --sql, --cut, or --pos option, data in all tables is imported in the specified format. To import all database object definitions and table data, you can specify the --all and --ddl options with a data format option.

  Notice

  The --all option is mutually exclusive with any database object options. It cannot be specified together with other database object options. If both the --all option and a database object option are specified, the --all option will be executed first.

• --table ' table_name [,table_name...] ' | --table ' * '

  Imports table definitions or table data. When this option is used with the --ddl option, only table definitions are imported. When this option is used with any data format option, only table data is imported. To specify multiple tables, separate the table names with commas (,). By default, for OceanBase Database in Oracle mode, the table names are in uppercase, and for OceanBase Database in MySQL mode, the table names are in lowercase. For example, for OceanBase Database in Oracle mode, both --table 'test' and --table 'TEST' indicate the table named TEST. For OceanBase Database in MySQL mode, both --table 'test' and --table 'TEST' indicate the table named test. If table names are case-sensitive, enclose them in brackets ([ ]). For example, --table '[test]' indicates the table named test, while --table '[TEST]' indicates the table named TEST. If the table name is specified as an asterisk (*), all table definitions or table data is imported.

  Notice

  When you use a control file to import data, the table name specified in the --table option must be in the same letter case as that in the database. Otherwise, the control file fails to take effect.

• --view ' view_name [, view_name...] ' | --view ' * '

  Imports view definitions. This option is similar to the --table option, except that this option does not support data import.
  
  

• --trigger ' trigger_name [, trigger_name...] ' | --trigger ' * '

  Imports trigger definitions. This option is similar to the --table option, except that this option does not support data import.
  
  

• --sequence ' sequence_name [, sequence_name...] ' | --sequence ' * '

  Imports sequence definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.
  
  

• --synonym ' synonym_name [, synonym_name...] ' | --synonym ' * '

  Imports synonym definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.
  
  

• --type ' type_name [, type_name...] ' | --type ' * '

  Imports type definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.
  
  

• --type-body ' typebody_name [, typebody_name...] ' | --type-body ' * '

  Imports type body definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.
  
  

• --package ' package_name [, package_name...] ' | --package ' * '

  Imports package definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.
  
  

• --package-body ' packagebody_name [, packagebody_name...] ' | --package-body ' * '

  Imports package body definitions. This option is similar to the --table option, except that this option does not support data import. This option is supported only for OceanBase Database in Oracle mode.
  
  

• --function ' function_name [, function_name...] ' | --function ' * '

  Imports function definitions. This option is similar to the --table option, except that this option does not support data import.
  
  

• --procedure ' procedure_name [, procedure_name...] ' | --procedure ' * '

  Imports stored procedure definitions. This option is similar to the --table option, except that this option does not support data import.
  
  

• --replace-object

  Replaces existing database object definitions during the import. For tables and synonyms, existing definitions are deleted and then new ones are created. For functions and stored procedures, existing definitions are replaced by using the CREATE OR REPLACE statement. This option can be used only with the --ddl or --mix option, and does not take effect for --csv, --sql, and other data format options.

  Notice

  • If an object already exists in the destination database, the object definition is forcibly replaced with the object definition stored in the import file.
  • If you do not need to replace objects in the destination database, do not use this option. Otherwise, business may be affected.

• --retry

  Resumes the import task from the breakpoint. We recommend that you specify this option to resume an import task that has more than 80% of data imported, so that you do not have to start a new import. Duplicate data errors may occur during the resumed import. If only a small part of the data has been imported, you can clear tables and start a new import, which is more efficient.

  Notice

  The CHECKPOINT.bin file is a savepoint file generated when the tool runs, and is located in the directory specified in the -f option. You cannot use this option if the CHECKPOINT.bin file does not exist.

• --external-data

  Specifies that the data files to be imported are exported by a third-party tool. OBDUMPER generates a MANIFEST.bin file in the directory specified in the -f option during data export. The file stores metadata information. When OBLOADER imports data, it parses the metadata file by default. If the metadata file is missing, or the data is exported by a third-party tool without a metadata file, you can specify this option to skip metadata parsing for the import.
  
  

• --nls-date-format ' date-format-string '

  The format of dates in database connections in Oracle mode of OceanBase Database. The default value is YYYY-MM-DD HH24:MI:SS.
  
  

• --nls-timestamp-format ' timestamp-format-string '

  The format of timestamps in database connections in Oracle mode of OceanBase Database. The default value is YYYY-MM-DD HH24:MI:SS: FF9.
  
  

• --nls-timestamp-tz-format ' timestamp-tz-format-string '

  The format of timestamps that contains a time zone in database connections in Oracle mode of OceanBase Database. The default value is YYYY-MM-DD HH24:MI:SS:FF9 TZR.
  
  

• --skip-header

  Skips headers of CSV/CUT files when the files are imported. Headers are data in the first line. This option can be used only with the --cut or --csv option. Data in the first line of CUT files can be skipped only in OBLOADER V3.3.0 and later.
  
  

• --skip-footer

  Skips footers of CUT files when the files are imported. Footers are data in the last line. This option can be used only with the --cut option.
  
  

• --null-string ' null_replacer_string '

  Replaces a specified character with NULL. This option can be used only with the --cut or --csv option. The default value is \N.
  
  

• --empty-string ' empty_replacer_string '

  Replaces a specified character with an empty string (' '). This option can be used only with the --csv option. The default value is \E.
  
  

• --line-separator ' line_separator_string '

  The line separator. Custom line separators are supported for the import of CUT files. The default value of this option is relevant with the system platform. The values available are \r, \n, and \r\n only.

  Note

  This option can be used only with the --cut or --csv option.

• --column-separator ' column_separator_char '

  The column separator. This option can be used only with the --csv option and supports a single character only. The default value is a comma (,).
  
  

• --escape-character ' escape_char '

  The escape character. This option can be used only with the --cut or --csv option and supports a single character only. The default value is a backslash ().
  
  

• --column-delimiter ' column_delimiter_char '

  The column delimiter. This option can be used only with the --csv option and supports a single character only. The default value is a single quotation mark (').
  
  

• --with-trim

  Deletes the space characters on the left and right sides of the data. This option can be used only with the --cut or --csv option.
  
  

• --trail-delimiter

  Truncates the last column separator in a line. This option can be used only with the --cut or --csv option.
  
  

• --ignore-unhex

  Ignores hexadecimal strings in decoding. This option applies only to binary data types.
  
  

• --exclude-table ' table_name [, table_name...] '

  Excludes the specified tables from the import of table definitions or table data. Fuzzy match on table names is supported, for example, --exclude-table 'test1,test*,*test,te*st' The preceding example specifies to exclude the following tables from the import of table definitions or table data:

  • test1
  • All tables with a table name starting with test
  • All tables with a table name ending with test
  • All tables with a table name starting with te and ending with st
    
    

• --exclude-data-types ' datatype [, datatype...] '

  Excludes the specified data types from the import of table data.
  
  

• --column-splitter ' split_string '

  The column separator. This option can be used only with the --cut option.
  
  

• --storage-uri ' storage_uri_string '

  The URI of the storage space. OBLOADER V4.2.0 and later support importing database object definitions and table data from Alibaba Cloud OSS or Amazon Simple Storage Service (S3). OBLOADER V4.2.1 and later also support importing database object definitions and table data from Hadoop.

  The syntax of 'storage_uri_string' is as follows:

  [scheme://host]path[?parameters]
  
  parameters: key[=value],...
  

  The following table describes the components.

  Component	Description
  scheme	The storage scheme. Alibaba Cloud OSS, Amazon S3, and Apache Hadoop are supported. If the scheme is not Alibaba Cloud OSS, Amazon S3, or Apache Hadoop, an error is returned.
  host_name	The name of the storage space. When you import data from Alibaba Cloud OSS or Amazon S3, the host parameter specifies a bucket. For more information, see OSS Bucket. When you import data from Apache Hadoop, the host parameter specifies a Hadoop node, which is in the <ip>:<port> or <cluster_name> format.
  path	The data storage path in the storage space. The path must start with a slash (/).
  parameters	The parameters required for the request. The value can be a single key or multiple key-value pairs.

  Example: Import data from Amazon S3

  --storage-uri 's3://bucket/path?region={region}&access-key={accessKey}&secret-key={secretKey}'
  

• s3: indicates that the storage scheme is Amazon S3.
• bucket: the name of the bucket in Amazon S3.
• path: the data storage path in the Amazon S3 bucket.
• ?region={region}&access-key={accessKey}&secret-key={secretKey}: the region, AccessKey ID, and AccessKey secret.

The following table describes the supported parameters.

• --max-discards int_num

  The maximum amount of duplicate data allowed in a single table to import. If duplicate data in a table exceeds the specified maximum, data import for this table stops. The import failure on the table is logged. Data import for other tables is not affected. The default value is -1, indicating that data import does not stop for duplicate data.

  Note

  This option takes effect only when the table contains a primary key or a unique key and duplicate data exists in the table.

• --max-errors int_num

  The maximum number of errors allowed for an import. If the number of import errors of a table exceeds the specified maximum, data import for this table stops. The import failure on the table is logged. This option can be set to 0, -1, or any positive integer N. If you set this option to -1, the errors are ignored and the import continues. The default value is 1000.
  
  

• --exclude-column-names ' column_name [, column_name...] '

  Notice

  • The letter case of the specified column name must be the same as that of the column name in the table structure.
  • In the imported data file, the excluded columns must have no corresponding data, and the imported columns must be in the same order as the columns in the table.

• --replace-data

  Replaces duplicate data in the table. This option is only applicable to tables that have primary keys or unique keys with the NOT NULL constraint. If a file has a large amount of duplicate data, for example, more than 30% of the total data volume, we recommend that you clear the table and import it again. The performance of data replacement is lower than that of import after table clearing. This option can be used only with the --cut, --csv, and --sql options, and does not take effect for the --ddl option.

  Notice

  • When the file and the table have duplicate data, the data in the table is replaced with the data in the file.
  • For tables without primary keys or unique keys, this option appends data to the tables.
  • If you do not need to replace duplicate data, do not specify this option. Otherwise, business may be affected.

• --truncate-table

  Truncates tables in the destination database. This option can be used only with a data format option. When being used with the --all or --table '*' option, this option specifies to truncate all tables in the destination database. If you want to truncate only some of the tables, you can explicitly specify them in the format of --table 'test1,test2,[....]'. When being used with the --with-data-files option, this option specifies to truncate tables that have the corresponding data files.

  Notice

  • If the --all or --table '*' option is used with the --truncate-table option, the tool truncates all tables in the destination database, even if no corresponding data files of the tables exist in the directory specified by the -f option.
  • Do not use this option to truncate the destination database or destination tables. We recommend that you manually truncate tables to avoid impacts on business.

• --with-data-files

  When being used with the --truncate-table or --delete-from-table option, this option specifies to truncate or clear tables that have corresponding data files before data import. This option does not take effect if it is used separately.
  
  

• --delete-from-table

  Clears tables in the destination database before the import. This option can be used only with a data format option. When being used with the --all or --table '*' option, this option specifies to clear all tables in the destination database. If you want to clear only some of the tables, you can explicitly specify them in the format of --table 'test1,test2,[....]'. When being used with the --with-data-files option, this option specifies to clear tables that have corresponding data files only.

  Notice

  • If the --all or --table '*' option is used with the --delete-from-table option, the tool clears all tables in the destination database, even if no corresponding data files of the tables exist in the directory specified by the -f option.
  • Do not use this option to clear the destination database or destination tables, especially tables with large data volumes. We strongly recommend that you delete table data manually based on your business requirements to avoid impacts on your business.

• --file-regular-expression

  The regular expression that is used to match the file name during an import. This option applies only to single-table import.
  
  

• --ignore-escape

  Ignores escape operations on characters for the import of CUT files. By default, escape operations are not ignored.
  
  

• --strict= ' strict_string '

  Specifies to control the impact of dirty data on the process end status during import. The default value is true, which indicates that the program is to end in the failed state (System exit 1) when the imported data contains a Bad Record or Discard Record error. The value false indicates that the final end state (System exit 0) of the program is not affected when the imported data contains a Bad Record or Discard Record error.

  Note

  This option can be used with the --max-discards or --max-errors option, which specifies to skip duplicate data or errors and proceed with the process when the amount of duplicate data or the number of errors is within the specified range. For more information, see Error handling.

• --character-set ' character_set_string '

  The character set for creating a database connection. The default value is the value specified by the session variable jdbc.url.character.encoding in the session.properties file. The value of the --character-set option overrides that of the jdbc.url.character.encoding variable. Supported character sets include binary, gbk, gb18030, utf16, and utf8mb4.

Performance options

• --rw float_num

  The proportion of data file parsing threads. The default value is 0.2. You can use this option with the --thread option to calculate the number of file parsing threads based on the following formula: Number of file parsing threads = Value of --thread × Value of --rw.
  
  

• --slow float_num

  The threshold for triggering a slow import. When the memory usage of OceanBase Database reaches 75%, OBLOADER slows down to prevent an excessively high memory usage on the database. The default value is 0.75.
  
  

• --pause float_num

  The threshold for triggering an import pause. When the memory usage of OceanBase Database reaches 85%, OBLOADER pauses data import to prevent issues caused by an excessively high memory usage on the database. The default value is 0.85.
  
  

• --batch int_num

  The number of records for a batch of transactions. We recommend that you set this option to a value inversely proportional to the table width. Do not set this option to an excessively high value, which may cause database memory overflow. The default value is 200.
  
  

• --thread int_num

  The number of concurrent export threads allowed. This option corresponds to the number of import threads. You can use this option with the --rw option to calculate the number of file parsing threads based on the following formula: Number of file parsing threads = Value of --thread × Value of --rw. The default value is Number of CPU cores × 2. OceanBase Database executes DDL statements in series. Therefore, you do not need to specify this option when you import database object definitions.
  
  

• --block-size int_num

  The block size for a file to be imported. When specifying this option, you do not need to explicitly specify the unit. The default unit is MB. By default, OBLOADER automatically splits a large file into multiple logical subfiles (or blocks) sized 64 MB. The logical subfiles do not occupy additional storage space. The default value is 64.
  
  

• --max-tps int_num

  The maximum import speed. You can specify this option to ensure import stability.
  
  

• --max-wait-timeout int_num

  The timeout period of waiting for a database major compaction to complete. When specifying this option, you do not need to explicitly specify the unit. The default unit is hour. When the database is under a major compaction, OBLOADER stops data import and waits up to the specified period. The default value is 3.

Other options

• -H, --help

  Shows the help information of the tool.
  
  

• -V, --version

  Shows the version of the tool.