Skip to main content
Flux synchronization is designed for durability. A singleton worker serializes all DuckDB writes while each data source maintains its own schedule and independent completion checkpoints. A failed scope does not discard successful sibling scopes, and a restarted worker recovers mid-flight work automatically once it acquires the released execution lease.

​
Worker modes

The sync worker mode is controlled by the FLUX_SYNC_WORKER_MODE environment variable: In both modes, the worker polls the request queue at the interval set by FLUX_SYNC_WORKER_POLL_SECONDS (default 5 seconds).

​
Sync request lifecycle

1

Enqueue

A scheduled trigger or an administrator calling POST /api/integrations/azure/sync writes a sync_runs record with status queued.
2

Claim

The singleton worker acquires the sync execution lease and transitions the request to running. Only one worker can hold the lease at a time. The lease duration is controlled by FLUX_SYNC_CLAIM_LEASE_SECONDS (default 900 seconds).
3

Collect

The worker collects each configured source and subscription/cost-type scope. Each scope attempt is recorded in sync_source_runs.
4

Checkpoint

After each scope completes successfully, sync_source_runs is updated with the completion state. source_sync_state advances only for completed scopes — a failed scope does not overwrite the previous good result.
5

Complete

When all scopes finish, the sync_runs record is marked complete.
Worker replacement recovery: if the worker process exits while a request is running, the OS releases the execution lease. The replacement worker picks up the same request once it acquires the lease, reads the per-scope checkpoints in sync_source_runs, skips already-completed scopes, and resumes only unfinished work.

​
Independent source schedules

Each source has its own scheduled enqueue time. These are production schedules:
An ad-hoc sync triggered via POST /api/integrations/azure/sync collects inventory, Advisor, Flux Intelligence, and Azure Policy only. It deliberately excludes Cost Management to avoid competing for QPU quota with the scheduled cost jobs.

​
Checkpoint and recovery

sync_source_runs records every per-source and per-subscription/scope attempt, including row counts, retry attempts, and last-good retention status. This table is the authoritative checkpoint:
  • Recovery skips completed scopes — a recovering worker reads the checkpoint and skips any scope that already succeeded.
  • Partial success is preserved — if a later scope fails, all previously checkpointed scopes are unaffected. Their data is already committed to DuckDB.
  • Retry ordering — missing and failed scopes run first; never-collected commitment scopes run ahead of healthy cost refreshes.

​
Last-good retention

source_sync_state advances only after a scope fully completes. This means:
  • A throttled subscription retains its previous good cost view.
  • A failed Flux Intelligence rule pack retains the previous complete finding set.
  • A partial initial cost-history scope remains retry-eligible rather than being promoted to rolling refresh.
If the Cost Management Query API persistently fails for a scope, an automated Cost Details fallback fills the affected calendar months asynchronously (up to FLUX_COST_DETAILS_MAX_REPORTS_PER_RUN reports per daily run) without extending the primary Azure sync.

​
Schedule-aware staleness

Flux does not declare a source stale just because data has not changed recently. A source is stale only when the last expected scheduled run has been missed plus a grace window:
  • Daily sources: 4-hour grace window after the scheduled time.
  • Interval sources: 2-hour grace window.
A hard backstop (stale_after_hours) still applies to sources without a defined schedule. Unprovisioned sources (never synced) do not trigger the staleness warning banner.

​
Administration page

The Administration page shows the following for every source:
  • Schedule (next expected run time)
  • Current row count
  • Completed vs expected scopes
  • Last attempt time and outcome
  • Whether last-good data is being served
The Overview and Opportunities pages warn when source evidence is stale or degraded.

​
Configuration reference