Incremental Sync in Automatic Exports

Chargebee supports Incremental Sync, enabling you to export only the changes (create, update, delete) that occurred since the last successful export. Incremental Sync supports efficient Change Data Capture (CDC) workflows and reduces data processing and transfer costs by eliminating redundant full exports.

How Incremental Sync Works

• First Export (Snapshot):
  The initial export for each dataset is a full snapshot. All records are included with change_type = read.
• Subsequent Exports (Incremental):
  Each subsequent export includes only the records that changed—created, updated, or deleted—since the previous export run.
• Metadata:
  The metadata column definitions remain consistent across snapshot and incremental sync. Each record includes:
  • change_type: Indicates the type of change (create, update, delete, read) and can be used to load data into the warehouse accordingly.
  • record_extracted_at: Timestamp when the updated data became available in Chargebee’s data lakehouse.
  • record_exported_at: Timestamp when the data was exported to the configured destination.
• Destination Folder: Both the first snapshot and incremental data will be delivered to the incremental_sync folder.

How to Use the Data

The data is available in year, month, day, and hour-level partitions. The hour corresponds to the UTC hour when the export was scheduled. For example:

s3://<your-bucket>/<your-subfolder>/<your-domain>/incremental_sync/data/invoices/year=2025/month=08/day=15/hour=03

Use the change_type field to determine the appropriate operation to perform in the destination system (for example, insert, update, or delete).

• Deleted records will have change_type = delete.
• Utilize record_extracted_at to sort the records.
• A schema file is also included for each dataset and updated when changes occur. This schema defines the structure of the underlying data. The schema file is available at the following path:

s3://<your-bucket>/<your-subfolder>/<your-domain>/incremental_sync/schema/<dataset-name>.json

Record Journey in Incremental Sync

Chargebee’s incremental sync tracks data changes over time, enabling downstream systems to maintain accurate, up-to-date datasets.

First Export (Snapshot)

The first export provides a snapshot of all existing records.

For example:

change_type = read indicates a snapshot record. The first export for every table is always a snapshot of the dataset.

Subsequent Exports (Incremental)

Subsequent exports provide incremental updates over the previous export.

Continuing the previous example, assume the following changes occur:

• John Doe’s last name is updated.
• Alice Brown has been deleted.
• Carlos Rivera’s last name was changed to ‘Young’ and was subsequently deleted.
• A new record for Maria Chen is created, updated, and deleted all at once.

The following table shows how the records appear in the next export:

• Only the latest change to each record within the export batch is included.
• No data for Maria Chen’s record is sent, because the record is created and subsequently deleted within the same export batch.

Scenario Handling

The following table illustrates how different scenarios are handled within Incremental Sync: