Incremental Sync in Automatic Exports

Chargebee supports Incremental Sync, enabling you to export only the changes (create, update, delete) that occurred since the last successful export. Incremental sync supports efficient Change Data Capture (CDC) workflows and reduces data processing and transfer costs by eliminating redundant full exports.

How Incremental Sync Works

• First Export (Snapshot):
  The initial export for each dataset is a full snapshot. All records are included with change_type = read.
• Subsequent Exports (Incremental):
  Each subsequent export includes only the records that changed—created, updated, or deleted—since the previous export run.
• Metadata:
  The metadata column definitions remain consistent across snapshot and incremental sync. Each record includes:
  • change_type: Indicates the type of change (create, update, delete, read) and can be used to load data into the warehouse accordingly.
  • record_extracted_at: Timestamp when the updated data became available in Chargebee’s data lakehouse.
  • record_exported_at: Timestamp when the data was exported to the configured destination.

How to Use the Data

The data is available in year, month, day, and hour-level partitions. The hour corresponds to the UTC hour when the export was scheduled. For example:

s3://<your-bucket>/<your-subfolder>/<your-domain>/incremental_sync/data/invoices/year=2025/month=08/day=15/hour=3

Use the change_type field to determine the appropriate operation to perform in the destination system (for example, insert, update, or delete).

• Deleted records will have change_type = delete.
• Sort by record_extracted_at to preserve the correct event order if multiple operations occur on the same row in a batch.
• A schema file is also included for each dataset and updated when changes occur. This schema defines the structure of the underlying data. The schema file is available at the following path:

s3://<your-bucket>/<your-subfolder>/<your-domain>/incremental_sync/schema/<dataset-name>.json

Record Journey in Incremental Sync

Chargebee’s incremental sync tracks data changes over time, enabling downstream systems to maintain accurate, up-to-date datasets.

Full Snapshot Sync (First Sync)

The first export provides a complete snapshot of all existing records:

change_type = read indicates a point-in-time snapshot. The first sync for every table is always a snapshot of the dataset (full export).

Incremental Sync (Subsequent Syncs)

Over time, the following changes occur:

• John Doe’s last name is updated.
• Alice Brown has been deleted.
• Carlos Rivera remains unchanged.
• A new record for Maria Chen is created, updated, and deleted all at once.

The following table shows how the records appear in the next export:

Schema Changes in Incremental Sync

Chargebee handles schema evolution gracefully: