Skip to main content

scoping

Running a DAG's scope step and rescoping the run to what it selects.

A scope step (produces_file_scope) narrows the run's file selection rather than producing a result other steps read. It runs once, before waves are planned, and its output reaches the run down two channels: the FileMetadataContext every step reads, and datasource.selected_file_names_override.

A scope step that keeps none of its files is not applied at all: an empty selected_file_names_override is falsy, and selected_file_names_iter reads falsy as no override and walks the whole datasource. apply_file_scope returns an EmptySelection instead, and the run ends without running a step.

Module​

Functions​

apply_file_scope​

def apply_file_scope(    *,    dag: BackgroundDAG,    cache: CacheProtocol,    context: FileMetadataContext,    datasource: Any,    task_hash: str,    project_id: str | None,    run_id: str | None,) ‑> EmptySelection | None:

Run dag's scope step, if it has one, and rescope the run to its output.

A no-op for a DAG that declares no scope step. A scope step that fails leaves the run unscoped rather than partly scoped.

Arguments

  • dag: The parsed background phase, with its step hashes stamped.
  • cache: The pod's background cache.
  • context: The run's file-metadata provider.
  • datasource: The datasource the steps share, or None.
  • task_hash: The datasource-level task hash.
  • project_id: The lineage's project, or None.
  • run_id: The flow run, for the step's telemetry.

Returns An EmptySelection when the scope step kept none of the files it was given, in which case nothing is applied; otherwise None.