Skip to main content

file_metadata

file_metadata runtime package.

Re-exports the public API so callers can import directly from bitfount.runtimes.file_metadata without knowing the internal module layout:

from bitfount.runtimes.file_metadata import (
    file_metadata_runtime,
    file_metadata_refresh,
    generate_prefect_task_hash,
)

Module​

Submodules​

Functions​

file_metadata_refresh​

async def file_metadata_refresh(staleness_days: int = 1) ‑> int:

Re-index stale file metadata across all pods.

Scans every pod's background_cache.db under ~/.bitfount/cache/pods/, identifies datasources with rows older than staleness_days, and submits a background/file_metadata_runtime deployment run for each one — unless a non-stale indexing run for that datasource is already active.

This flow is intended to be registered as a single long-lived Prefect deployment via file_metadata_refresh.serve(cron=...) on orchestrator startup. It uses global_limit=1 so only one refresh pass runs at a time even if a scheduled tick fires while the previous pass is still running.

Runs are submitted as fire-and-forget (timeout=0) so the refresh flow returns as soon as all runs are scheduled, not when they complete. is_flow_run_active (a Prefect flow-run tag query) prevents duplicate runs from being scheduled for the same datasource; _is_run_active is still called per datasource but only reaps orphaned bookkeeping rows.

Arguments

  • staleness_days: Number of days after which a FileMetadata is considered stale and eligible for re-indexing. Defaults to 1. Because a re-index rewrites indexed_at on every row of the datasource, no row is stale again until this window elapses — so this value, not the deployment's cron, is what sets the re-walk cadence. Ticking more often than the window does nothing.

Returns The total number of file_metadata_runtime runs triggered.

file_metadata_runtime​

def file_metadata_runtime(    pod_name: str,    datasource_name: str,    datasource_type: str,    cache_db_path: str,    orchestrator_callback_url: str | None = None,    path: str | None = None,    connection_string: str | None = None,    only_paths: list[str] | None = None,) ‑> int:

Collect and persist file metadata for a single datasource.

This flow is triggered as a fire-and-forget background task immediately after a pod is successfully started (i.e. the dataset has been created). The orchestrator submits it via run_deployment("background/file_metadata_runtime", timeout=0) so it never blocks the HTTP response.

A Run is written at the start (status "running") and updated on completion or failure. The refresh flow checks this record before re-triggering to avoid concurrent indexing passes for the same datasource.

On completion the flow POSTs a file_metadata_complete payload to orchestrator_callback_url (when provided) so the orchestrator can forward the event to the desktop app via Socket.IO. The notification is sent whether the flow succeeds or fails.

Datasource routing uses issubclass checks against the resolved class so that newly added subclasses are handled automatically without any changes to this module:

  • FileSystemIterableSource subclasses → directory walk.
  • _SQLSource subclasses → DB connection string row.
  • Everything else with a path → single-file metadata (e.g. CSV).
  • Unresolvable or unsupported types → logged and skipped.

Arguments

  • pod_name: Name of the pod (used to derive task_hash).
  • datasource_name: Name of the datasource within the pod.
  • datasource_type: Datasource type string from the pod config, e.g. "bitfount.CSVSource" or "bitfount.DICOMSource".
  • cache_db_path: Filesystem path to the pod's background_cache.db SQLite file. The flow opens its own cache connection for the duration of the run and closes it on exit.
  • orchestrator_callback_url: Optional URL of the orchestrator's POST /pod/file-metadata-complete endpoint. When set, the flow POSTs a completion payload so the orchestrator can emit a file_metadata_complete Socket.IO event to the desktop app.
  • path: File path (single-file sources) or directory path (FileSystemIterableSource subclasses).
  • connection_string: DB connection string (_SQLSource subclasses).
  • only_paths: When set, index exactly these paths (files or directories) instead of walking path, and skip the deletion prune. Used by the file watcher, which has already identified what is new. Ignored for non-FileSystemIterableSource datasources, which have no walk to scope. Any spelling is accepted — these and path are both normalized before being compared — and a path that lands outside path is refused with a warning. See bitfount.runtimes.file_metadata.tasks._iter_stat_targets.

Returns The number of FileMetadata records written to the cache.

generate_prefect_task_hash​

def generate_prefect_task_hash(pod_name: str, datasource_name: str) ‑> str:

Return a stable, fixed-length task hash for a (pod, datasource) pair.

The hash is derived from "{pod_name}/{datasource_name}" so that it is deterministic across restarts and serves as the composite primary-key prefix in FileMetadata and Run.

This is the datasource leaf hash, and it plays two roles:

  • It keys the datasource/run-level tables directly — file_metadata, scan_metadata, runs, schema_versions — none of which a DAG step writes.
  • It is the leaf mixed into every per-step Merkle task_hash (bitfount.flows.dag.hashing), which is what keeps two datasources on the same pod in separate partitions of a step's table.

Arguments

  • pod_name: The name of the pod.
  • datasource_name: The name of the datasource within the pod.

Returns A 32-character hex string (the first 128 bits of the SHA-256 digest).