Skip to main content

models

Data models for the background DAG.

These are the core primitives that represent a parsed and validated v9 background DAG. They are pure data — no I/O, no registry access.

Input references​

Two kinds of reference can appear in a step's inputs: block:

FromRef Refers to a field on the result of a previously executed step. Syntax in YAML: step_name.field e.g. ga_inference.cache

ContextRef Refers to a value provided by a named ContextProvider (see context.py). The $ prefix in YAML is stripped when parsing. Syntax in YAML: $provider_name.field e.g. $file_metadata.cache

BackgroundRef Used only in interactive DAGs. Refers to a field on the result of a background step that ran in a separate execution and whose output lives in the cache. Same YAML syntax as FromRef (step.field); the parser emits a BackgroundRef instead when the referenced step is a background step. Resolved at runtime via the BackgroundResultsContext provider.

Classes​

BackgroundDAG​

class BackgroundDAG(**data: Any):

A fully parsed and validated background DAG ready for execution.

Attributes

  • name: Human-readable name of the flow (from FlowSpec.name).
  • federation_strategy: The federation strategy declared in the YAML (e.g. worker_only).
  • steps: Ordered list of steps. Execution order is declaration order; parallel steps are submitted as futures and flushed on demand.
  • extra: Any additional top-level metadata from the FlowSpec that callers may find useful (e.g. for logging or telemetry).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Variables​

  • static federation_strategy : str
  • static model_config
  • static name : str

BackgroundRef​

class BackgroundRef(**data: Any):

A cross-phase reference from an interactive step to a background step.

Background steps run in a separate execution; their outputs are read back from the cache at interactive-DAG runtime via BackgroundResultsContext.

Example YAML (inside an interactive step): ga_inference.cache

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Variables​

  • static model_config
  • static output_field : str - Attribute name on the background step's result object (e.g. cache).
  • static step : str - Name of the background step whose result is referenced.

ContextRef​

class ContextRef(**data: Any):

A reference to a value provided by an ambient ContextProvider.

Example YAML: $file_metadata.cache

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Variables​

  • static field : str - Field name to resolve from the provider.
  • static model_config
  • static provider : str - Name of the context provider (e.g. file_metadata).

DAGRunContext​

class DAGRunContext(**data: Any):

Runtime context for a single background DAG execution.

Bundles everything the executor needs beyond the DAG itself so functions take (dag, run_ctx) instead of four separate parameters.

Attributes

  • context_providers: Mapping of provider name → ContextProvider used to resolve ContextRef inputs (e.g. file_metadata).
  • runtime_params: Runtime objects injected into every step's kwargs as a base layer (e.g. datasource, cache, task_hash). YAML-resolved refs override any overlapping keys.
  • lifecycle_notifier: Signals DAG lifecycle transitions (accept / success / failure) to whichever destinations are wired (cache Run row, initiator mailbox). Defaults to a no-op notifier.
  • background_results: Provider used to resolve BackgroundRef inputs (cross-phase references from interactive steps to background step outputs). None for background DAGs, which have no such refs.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Variables​

  • static model_config

DAGStep​

class DAGStep(**data: Any):

A single step in the background DAG — one node to execute.

Attributes

  • name: Unique step name within the DAG (e.g. fovea_inference).
  • task: Registry key for the step (e.g. model_inference).
  • version: Step version to look up in the registry.
  • config: Instantiated Pydantic config for this step, or None if the step has no config class or no config was provided.
  • inputs: Mapping of parameter name → resolved InputRef.
  • parallel: If True the step is submitted as a Prefect future and its result is not awaited until a downstream step needs it.
  • save_to_cache: Declarative hint from the YAML. Parsed and stored but not currently acted upon — steps handle their own caching.
  • task_hash: This step's cache partition key — a Merkle hash over its own semantic config and its upstream steps' hashes, assigned by hashing.stamp_step_task_hashes when the run context is assembled. None until then (parsing and validation do not need it, and neither does a caller that only inspects the DAG); the executor then falls back to the run's datasource-level runtime_params["task_hash"].
  • parent_task_hashes: The task_hash of each upstream step this step declares an input from, keyed by this step's input parameter name — its routing table for reading an upstream partition through a direct store call, which a CacheAccessor cannot express. Assigned alongside task_hash; None until then. Keyed by parameter rather than upstream step name because the parameter is part of this step's own contract, while the upstream step's name is a label the flow author may change. Not part of the hashed payload (see hashing), so its contents never affect any partition key.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Variables​

  • static config : pydantic.main.BaseModel | None
  • static model_config
  • static name : str
  • static parallel : bool
  • static parent_task_hashes : dict[str, str] | None
  • static save_to_cache : bool
  • static task : str
  • static task_hash : str | None
  • static version : int

FromRef​

class FromRef(**data: Any):

A reference to a field on the result of a prior step.

Example YAML: ga_inference.cache

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Variables​

  • static model_config
  • static output_field : str - Attribute name on the step's result object.
  • static step : str - Name of the step whose result is referenced.

InteractiveDAG​

class InteractiveDAG(**data: Any):

A fully parsed and validated interactive DAG ready for execution.

The interactive DAG is the post-processing phase of a v9 flow (GA calculation, criteria matching, report generation). It runs as a separate execution from the background DAG and may reference background step outputs via BackgroundRef inputs, which are resolved from the cache at runtime.

Attributes

  • name: Human-readable name of the flow (from FlowSpec.name).
  • federation_strategy: The federation strategy declared in the YAML (e.g. worker_only).
  • steps: Ordered list of interactive steps. Execution order is declaration order; parallel steps are submitted as futures and flushed on demand.
  • background_steps: The parsed background DAGStep list (carrying each step's instantiated config). Used to build the BackgroundResultsContext and to validate BackgroundRef targets; not executed by the interactive run.
  • extra: Any additional top-level metadata from the FlowSpec.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Variables​

  • static background_steps : list[DAGStep]
  • static federation_strategy : str
  • static model_config
  • static name : str
  • background_step_names : frozenset[str] - Names of the background steps interactive steps may reference.