Skip to main content

deployments

Deployment identity and readiness for the background metadata runtimes.

Both metadata flows register under the same Prefect flow name (background) with distinct deployment names, and are addressed by the "<flow-name>/<deployment-name>" strings run_deployment accepts. Those strings are load-bearing in four places — the flows' own serve() calls, the orchestrator's triggers and chain automation, run_pod's triggers, and the refresh flow — so they live here rather than being retyped at each site: a mismatched literal does not fail loudly, it silently produces a deployment nothing ever triggers.

wait_for_deployment lives here for the same reason. serve() registers a deployment asynchronously from a daemon thread, so any code that triggers a run may get there first; every serving process therefore needs the same poll-until-registered probe.

Also home to the run_type values every flow run this system creates is tagged with, and the map from those to the deployment each is submitted against. They sit beside the handles because they are the same kind of thing — identity every submitter, every dedup check and both recovery sweeps must agree on — and because what has to hold of them is mutual: they must all be distinct, and in particular none may collide with BACKGROUND_DAG_RUN_TYPE, since flows.dag.recovery finds its work by matching that tag and would otherwise start replaying metadata runs it has no spec for. That invariant is checkable at a glance in one list and unverifiable spread across the runtime packages, which is also why BACKGROUND_DAG_RUN_TYPE lives here despite having no deployment: it is a run_type, and this is where run types are.

Also home to ConfigError, the misconfiguration signal both flows raise (see the class docstring for why the type matters).

Module​

Functions​

wait_for_deployment​

async def wait_for_deployment(    deployment_name: str, timeout: float = 30.0, poll_interval: float = 0.2,) ‑> bool:

Poll the Prefect API until deployment_name is registered.

Intended to be awaited by a process that has just started serve() in a background thread, before it submits any run. serve() registers the deployment asynchronously, so a triggerer that does not wait can call run_deployment against a deployment that does not exist yet.

Coroutine rather than a blocking function so the caller owns the event loop: the orchestrator drives it through its _run_async bridge (Flask-SocketIO runs handlers on plain threads) and run_pod through asyncio.run. Probing inside one loop also avoids standing up and tearing down an event loop on every poll for the whole timeout.

Arguments

  • deployment_name: The full "<flow-name>/<deployment-name>" string, e.g. FILE_METADATA_DEPLOYMENT.
  • timeout: Maximum seconds to wait before giving up.
  • poll_interval: Seconds between consecutive API probes.

Returns True when the deployment was found within timeout seconds, False if the timeout was reached first. Never raises: a caller that cannot confirm readiness should degrade (skip indexing) rather than abort startup, so the failure is reported by return value.

Global variables​

  • BACKGROUND_DAG_RUN_TYPE - run_type value marking an in-process background DAG run. Distinct from the metadata runtimes' values so the two never dedup against each other, and so neither recovery sweep ever matches the other's work.
  • DEPLOYMENT_BY_RUN_TYPE : dict[str, str] - The deployment handle each metadata run_type is served and resubmitted against. Pure identity, so it lives with the handles rather than with the recovery sweep that reads it.
  • DEPLOYMENT_POLL_INTERVAL_SECONDS - Gap between consecutive registration probes.
  • DEPLOYMENT_READY_TIMEOUT_SECONDS - How long a triggerer waits for a deployment to appear before giving up.
  • FILE_METADATA_DEPLOYMENT - Fully-qualified "<flow-name>/<deployment-name>" handles accepted by run_deployment and read_deployment_by_name. The orchestrator and run_pod must agree on these: a live run tagged by either triggerer is only visible to the other's dedup check if both scope it to the same deployment.
  • FILE_METADATA_DEPLOYMENT_NAME - Deployment names as passed to flow.serve(name=...).
  • FILE_METADATA_REFRESH_RUN_TYPE - run_type values for the remaining flow runs a pod produces. None of them are dedup keys — they exist so that no row in the Prefect UI is unlabelled, which is what makes the UI usable for support. Each is distinct from BACKGROUND_DAG_RUN_TYPE so that flows.dag.recovery never matches them.
  • FILE_METADATA_RUN_TYPE - The metadata runtimes' own run_type values. Named here rather than spelled as literals at each submission site, because bitfount.runtimes.recovery discovers lost runs by these values and a typo at one submitter would make those runs silently unrecoverable.
  • FLOW_NAME - Prefect flow name shared by file_metadata_runtime and scan_metadata_runtime. They coexist under it because deployment names, not flow names, disambiguate them.

Classes​

ConfigError​

class ConfigError(*args, **kwargs):

Raised when a metadata runtime's datasource configuration is invalid.

A dedicated type (rather than a bare ValueError) is what lets each flow's except Exception handler call mark_run_failed, store the message, and re-raise unconditionally — so Prefect records the run as failed rather than successful. Operators watching Prefect flow state then get an accurate signal without inspecting the cache DB or a callback payload.

Subclasses ValueError rather than BitfountError: the misconfiguration is always a bad argument to the flow (an unsupported datasource type, a missing path or connection string), and callers that already catch ValueError around flow invocation should keep catching it.