CREATE EXTERNAL TABLE|V4.3.3|OceanBase Database| docs|Distributed Database

Description

This statement creates an external table in a database.

The external table is a key feature in a database management system. Generally, the data of a table in a database is stored in the storage space of the database, while an external table has data stored in an external storage service.

When you create an external table, you must specify the file path and file format of the data. Then, you can read the data of the external table from the external storage service. An external table is read-only. You can use it in a query statement, but you cannot perform Data Manipulation Language (DML) operations on it. You cannot define constraints or create indexes on an external table.

Syntax

CREATE EXTERNAL TABLE <table_name>
    ( [ <column_name> <column_type> [AS <expr>] ]
      [ , <column_name> <column_type> [AS <expr>] ]
      [ , ... ] )
    LOCATION = '<string>'
    formatTypeOptions
    [ PARTITION BY ( <column_name> [, <column_name> ... ] ) ]
    [ PARTITION_TYPE = USER_SPECIFIED ]
    [ PATTERN = '<regex_pattern>' ]
    [AUTO_REFRESH = 'xx']
formatTypeOptions:
    FORMAT = (
      TYPE = 'CSV'
      LINE_DELIMITER = '<string>' | <expr>
      FIELD_DELIMITER = '<string>' | <expr>
      ESCAPE = '<character>' | <expr>
      FIELD_OPTIONALLY_ENCLOSED_BY = '<character>' | <expr>
      ENCODING = 'charset'
      NULL_IF = ('<string>' | <expr>, '<string>' | <expr> ...)
      SKIP_HEADER = <int>
      SKIP_BLANK_LINES = { TRUE | FALSE }
      TRIM_SPACE = { TRUE | FALSE }
      EMPTY_FIELD_AS_NULL = { TRUE | FALSE }
    )
    | FORMAT = (TYPE = 'PARQUET')

Parameters

Parameter	Description
table_name	The name of the external table to be created.
column_name	The name of a column in the external table. By default, the data columns in the file are automatically mapped to the columns defined in the external table.
column_type	The type of a column in the external table. Constraints such as `DEFAULT`, `NOT NULL`, `UNIQUE`, `CHECK`, `PRIMARY KEY`, and `FOREIGN KEY` cannot be defined.
AS	Manually specifies column mappings. If the order of columns in the file does not match that in the external table, you can use the pseudo-column `metadata$filecol{N}` to specify the mapping between the Nth column in the file and a column in the external table. For example, `c2 INT AS (metadata$filecol4)` indicates that the `c2` column in the external table corresponds to the fourth column in the file. Note that if you manually specify column mappings, automatic mappings will become invalid, and you must manually define mappings for all columns.
LOCATION	The path where the external table files are stored. Generally, the data files of an external table are stored in a separate directory, which can contain subdirectories. When you create an external table, the system automatically collects all files in the specified directory. The local `LOCATION` parameter is specified in the format of `LOCATION = '[file://] local_file_path'`. `local_file_path` can be a relative path or an absolute path. If you enter a relative path, the current directory must be the installation directory of OceanBase Database. `secure_file_priv` specifies the path that an OBServer node has the permission to access. `local_file_path` must be a subdirectory of `secure_file_priv`. The remote `LOCATION` parameter is specified in the format of `LOCATION = '{oss\\|cos\\|s3}://$ACCESS_ID:$ACCESS_KEY@$HOST/remote_file_path'`. When you access an object storage service (OSS, COS, or S3), you must specify the access information, such as `$ACCESS_ID`, `$ACCESS_KEY`, and `$HOST`. These sensitive access information will be encrypted and stored in the system tables of the database. Notice When you use object storage paths, separate the parameters in the object storage path with `&`. Make sure that the characters in the values of the parameters allowed, which include uppercase and lowercase letters, digits, `/-_$+=`, and wildcard characters. If you enter characters other than the ones allowed, the setting may fail.
FORMAT = ( TYPE = 'CSV'... )	Specifies the CSV format for external files. `LINE_DELIMITER`: specifies the line delimiter for CSV files. The default value is `LINE_DELIMITER='\n'`. `FIELD_DELIMITER`: specifies the field delimiter for CSV files. The default value is `FIELD_DELIMITER='\t'`. `ESCAPE`: specifies the escape character for CSV files. It can only be 1 byte in length. The default value is `ESCAPE ='\'`. `FIELD_OPTIONALLY_ENCLOSED_BY`: specifies the characters that enclose the field values in CSV files. The default value is an empty string. `ENCODING`: specifies the character set encoding format of the file. For more information about the character sets supported in Oracle mode, see Character set and collation. The default value is UTF8MB4 if no character set is specified. `NULL_IF`: specifies the strings to be treated as `NULL` values. The default value is an empty string. `SKIP_HEADER`: specifies to skip the file header, and specifies the number of lines to skip. `SKIP_BLANK_LINES`: specifies whether to skip blank lines. The default value is `FALSE`, which specifies not to skip blank lines. `TRIM_SPACE`: specifies whether to remove leading and trailing spaces from fields in the file. The default value is `FALSE`, which specifies not to remove leading and trailing spaces from fields in the file. `EMPTY_FIELD_AS_NULL`: specifies whether to treat empty strings as `NULL` values. The default value is `FALSE`, which specifies not to treat empty strings as `NULL` values.
FORMAT = ( TYPE = 'PARQUET'... )	Specifies the PARQUET format for external files.
PATTERN	Specifies a regular pattern string to filter files in the `LOCATION` directory. For each file in the directory specified by `LOCATION`, the external table accesses the file if the file path matches the pattern string, and skips the file if the file path does not match the pattern string. By default, if this parameter is not specified, the external table accesses all files in the directory specified by `LOCATION`. The external table stores the list of files that match the `LOCATION` parameter-specified path and are specified by `PATTERN` in the system tables of the database. During a scan, the external table accesses external files based on this list.
PARTITION_TYPE = USER_SPECIFIED	You must specify `PARTITION_TYPE = USER_SPECIFIED` if you want to manually add and drop partitions instead of letting the external table automatically manage partitions.
AUTO_REFRESH = 'xx'	Specifies how the system automatically refreshes the external table. Valid values are as follows: `OFF`: the default value, which specifies to disable automatic refresh. `INTERVAL`: allows you to manage automatic refresh rules for this external table by using `dbms_external_table`. `IMMEDIATE`: specifies to refresh the meta data of this SQL statement-related external table every time the SQL statement accesses the external table.

Usage notes

If an external file is deleted, the file no longer exists in the file list when you access the file list. In this case, the external table ignores the missing file.
If an external file is modified, the external table accesses the latest data of the external file. If the modification of the external file is concurrent with a query on the external table, the result of the query may be unexpected. Therefore, avoid modifying the external file while querying the external table.
The external table accesses only the files listed in the file list when new files are added to the external directory. If you want to add new files to the file list of the external table, you need to update the file list of the external table.

Examples

Set the path of secure_file_priv to /home/admin/, and place the extdata.csv file, which corresponds to the external table data to be imported, in the /home/admin/test directory on the local OBServer node that is connected.

Here is an example of setting the global secure path.
```
obclient> SET GLOBAL secure_file_priv = ""
Query OK, 0 rows affected
obclinet> \q
Bye
```
Note

After you set the secure_file_priv variable, you must execute the \q statement to quit and make the setting take effect.

The content of the CSV file is as follows:
```
1,'Dave','Smith','dsmith@outlook.com','friend',32
2,'Xena','Johnson','xjonson@outlook.com','contact',45
3,'Fred','Jackon','fjackson@outlook.com','co-worker',19
4,'Alma','Tyler','atyler@outlook.com','friend',53
```

obclient> CREATE EXTERNAL TABLE contacts (
    id    INT,
    firstname  VARCHAR(100),
    lastname   VARCHAR(100),
    email      VARCHAR(255),
    category   CHAR(30),
    age        NUMBER )
   LOCATION = '/home/admin/test/'
   FORMAT = (
     TYPE = 'CSV'
     FIELD_DELIMITER = ','
     FIELD_OPTIONALLY_ENCLOSED_BY =''''
    )PATTERN ='extdata.csv';

Query the data in the contacts external table.

obclient> SELECT * FROM contacts;
+------+-----------+----------+----------------------+--------------------------------+------+
| ID   | FIRSTNAME | LASTNAME | EMAIL                | CATEGORY                       | AGE  |
+------+-----------+----------+----------------------+--------------------------------+------+
|    1 | Dave      | Smith    | dsmith@outlook.com   | friend                         |   32 |
|    2 | Xena      | Johnson  | xjonson@outlook.com  | contact                        |   45 |
|    3 | Fred      | Jackon   | fjackson@outlook.com | co-worker                      |   19 |
|    4 | Alma      | Tyler    | atyler@outlook.com   | friend                         |   53 |
+------+-----------+----------+----------------------+--------------------------------+------+
4 rows in set

References

Create an external table

Manage external files

Update the file list of an external table

OceanBase

Customer Stories

Documentation

CREATE EXTERNAL TABLE