Skip to main content
Flux provides automated DuckDB backup to Azure Blob Storage and supports recovery at startup. After each successful sync the worker checkpoints the DuckDB file and uploads it to the configured Blob container. In the event of database corruption or accidental data loss, Flux can automatically restore the latest valid backup when it starts up.

​
Backup configuration

Set the following environment variables to enable backup. The managed identity assigned to the App Service must have Blob Data Contributor (or equivalent write) access to the target container.
Backup uses DefaultAzureCredential. On App Service this resolves to the system-assigned or user-assigned managed identity. For a user-assigned identity, also set FLUX_MANAGED_IDENTITY_CLIENT_ID.

​
When backups run

After each successful sync the worker:
  1. Issues a CHECKPOINT to flush the DuckDB WAL to the main file.
  2. Copies the checkpointed .duckdb file to a temporary directory.
  3. Uploads the copy to the configured Blob container under the prefix duckdb/ with the naming pattern flux-<YYYYMMDDTHHmmssZ>.duckdb.
  4. Prunes any blobs under duckdb/flux- whose last_modified timestamp is older than FLUX_BACKUP_RETENTION_DAYS.
The backup blob is uploaded with overwrite=False. Each sync produces a distinct timestamped blob rather than overwriting a single slot.

​
Analytics snapshot architecture

For multi-instance deployments, Flux supports an analytics snapshot tier that separates the singleton writer from read-only web-process consumers: In snapshot mode the web process is a read-only consumer; the singleton writer publishes immutable snapshots after each successful job. This eliminates the cross-process DuckDB write lock contention for multi-instance App Service plans.
The analytics snapshot daily retention tier (FLUX_ANALYTICS_SNAPSHOT_DAILY_RETENTION_DAYS) is separate from the legacy per-sync backup controlled by FLUX_BACKUP_*. When snapshot publishing is enabled, the per-UTC-day snapshot tier serves as the analytical backup.

​
Startup recovery

Set FLUX_RECOVER_DATABASE_FROM_LATEST_BACKUP=true to trigger automatic recovery at startup. When this flag is set, Flux:
1

Validate current database

Flux opens the existing database read-only and checks that the minimum required tables (azure_integration, resource_snapshots) are present and that resources_current contains at least one row. If the database is valid, no recovery is performed.
2

Identify candidates

If validation fails, Flux lists up to 8 backup blobs from the container (prefix duckdb/flux-, non-zero size), sorted by last_modified descending.
3

Download and validate

Flux downloads each candidate to a temporary file and runs the same validation check. Candidates that fail validation are discarded and the next one is tried.
4

Atomic replacement

Once a valid backup is found, Flux:
  • Acquires the cross-process writer lock (up to 180-second timeout).
  • Renames the damaged database to flux.duckdb.corrupt-<timestamp> (preserving it for post-mortem).
  • Renames any .wal file alongside it.
  • Atomically renames the validated backup into place.
5

Log outcome

Flux logs the restored backup name, the path of the preserved damaged file, and the resource count from the restored database.
Recovery is refused if the database file does not exist at FLUX_DUCKDB_PATH. Recovery restores only when the existing file fails validation — it does not replace a healthy database.

​
Manual restore

If you need to restore a specific backup outside of the automatic startup flow:
  1. Download the desired backup blob from the duckdb/ prefix in your backup container.
  2. Stop the Flux App Service (or all processes that hold the DuckDB writer lock).
  3. Replace the file at FLUX_DUCKDB_PATH (default data/flux.duckdb) with the downloaded backup.
  4. Remove any stale .wal file alongside it.
  5. Restart Flux.
Test database restoration regularly before relying on backups in production. Verify that the restored database passes the Flux health check at GET /api/health and that the Administration page shows expected source row counts.

​
Identity requirements

Recovery uses ManagedIdentityCredential directly (not DefaultAzureCredential). Ensure the managed identity has at minimum read access to the backup container for the recovery path, and write access for the backup upload path.