Incomplete data in batch export of multiple tables

Symptom

In a batch export of multiple tables in ODC, the data exported is incomplete

Possible causes

• The files to be exported have exceeded the size limit. Web ODC has a size limit for files to be imported and exported. If the file size exceeds the limit, data imported or exported may be incomplete.

  • Size limit for files to be imported and exported in ODC

    Operation	Versions earlier than ODC 3.2.0	ODC 3.2.0 and later versions
    Import	250 MB (after compression)	2 GB (after compression)
    Export	250 MB (before compression)	2 GB (before compression)

  • To check the file size, decompress the exported file and check whether the total size after decompression exceeds the corresponding size limit based on the ODC version.

• Empty tables or empty partitions exist in the tables to be exported.

  During export, the OBDUMPER component in ODC allocates the capacity limit evenly to each table, or evenly to each partition in a partitioned table. If an empty table or partition exists in the tables to be exported, empty tables or partitions with no data to export will cause the total size of the exported files to exceed the limit. The more empty tables or partitions, the more the size limit is exceeded.

  You can view the logs of the export task to check whether the size limit is triggered. For example, the following message prompts "Stop dump: ${schema}.${table} Size: ${actual size} >= ${expected size}"

  

Solutions

• Do not use ODC to export multiple tables at a time, especially when a large number of empty tables or partitions are involved. If data loss occurs in the export of a table and data truncation is caused by a size limit, export the table separately.

• Use OBDUMPER to batch export multiple tables.