refresh

Prefect flow for periodic re-indexing of stale file metadata.

This module implements the cron-driven counterpart to file_metadata_runtime. While the initial indexing is triggered once by the orchestrator when a pod is created, metadata rows grow stale over time as files are added, modified or removed. file_metadata_refresh re-indexes any datasource whose rows have not been updated within a configurable staleness window.

Design

A single long-lived flow.serve() call is started in a daemon thread at orchestrator startup. It registers one deployment on the embedded Prefect server and polls for scheduled work according to the configured cron expression. Because there is only one deployment (not one per datasource), no lifecycle management is required when pods are created or deleted.

At each scheduled tick the flow:

1. Scans ~/.bitfount/cache/pods/ for pod directories that contain a background_cache.db file.
2. For each DB, queries FileMetadata for distinct (pod_name, datasource_name, datasource_type, task_hash) groups where any row's indexed_at is older than staleness_days.
3. Skips any datasource for which a live (non-terminal) Prefect flow-run is already tagged with its task_hash — this prevents a second indexing pass from starting while the previous one (or the initial start_pod-triggered run) is still in progress. _is_run_active is still called per datasource, but only for its side effect of reaping orphaned "running" bookkeeping rows (decision: Prefect owns liveness/dedup, the runs table is bookkeeping-only cleanup, not a gate).
4. Submits a background/file_metadata_runtime deployment run for each stale datasource via run_deployment(..., timeout=0), tagged with task_hash:<task_hash> and run_type:file_metadata so future dedup checks (here and at the orchestrator's initial trigger) see it. Using the deployment path (rather than calling the flow directly as a sub-flow) ensures the same concurrency limits and deployment-level configuration that apply to orchestrator-triggered runs also apply here.

Input reconstruction

Because the provenance columns (pod_name, datasource_name, datasource_type, source_path) were written to every FileMetadata at index time, the refresh flow never needs to re-read the pod config JSON. The original input path or connection string is read directly from the source_path column; the legacy os.path.commonpath derivation is only used for rows indexed before that column was added.

Module

Functions

file_metadata_refresh

async def file_metadata_refresh(staleness_days: int = 1) ‑> int:

Re-index stale file metadata across all pods.

Scans every pod's background_cache.db under ~/.bitfount/cache/pods/, identifies datasources with rows older than staleness_days, and submits a background/file_metadata_runtime deployment run for each one — unless a non-stale indexing run for that datasource is already active.

This flow is intended to be registered as a single long-lived Prefect deployment via file_metadata_refresh.serve(cron=...) on orchestrator startup. It uses global_limit=1 so only one refresh pass runs at a time even if a scheduled tick fires while the previous pass is still running.

Runs are submitted as fire-and-forget (timeout=0) so the refresh flow returns as soon as all runs are scheduled, not when they complete. is_flow_run_active (a Prefect flow-run tag query) prevents duplicate runs from being scheduled for the same datasource; _is_run_active is still called per datasource but only reaps orphaned bookkeeping rows.

Arguments

• staleness_days: Number of days after which a FileMetadata is considered stale and eligible for re-indexing. Defaults to 1. Because a re-index rewrites indexed_at on every row of the datasource, no row is stale again until this window elapses — so this value, not the deployment's cron, is what sets the re-walk cadence. Ticking more often than the window does nothing.

Returns The total number of file_metadata_runtime runs triggered.

---