Skip to main content

run_background_task

Run a background DAG task from a FlowSpec YAML file.

This script is the CLI counterpart of the DATASET_PROJECT_LINKED gRPC trigger. It loads a v9 FlowSpec YAML, builds the background DAG, and executes it locally using the file metadata cache that was populated when run_pod started.

Prerequisites​

  • The pod must have been started at least once via bitfount run_pod so that the background cache (~/.bitfount/cache/pods/<pod_name>/background_cache.db) is populated with file metadata.

Usage​

bitfount run_background_task path/to/flow_spec.yaml \\
    --pod-config path/to/pod_config.yaml \\
    --datasource-name my-dataset \\
    [--project-id my-project-001]

--pod-config is the same YAML passed to bitfount run_pod; it is used to materialise the datasource and schema the inference steps need.

If --project-id is omitted a local run ID is generated. You must provide a project ID when the DAG includes steps that require model access gated by project.

Module​

Functions​

run​

def run(    path_to_task_yaml: str | PathLike[str],    pod_config: str | PathLike[str],    datasource_name: str,    project_id: str | None = None,) ‑> None:

Run a background DAG from a FlowSpec YAML file.

Arguments

  • path_to_task_yaml: Path to the v9 FlowSpec YAML file.
  • pod_config: Path to the pod config YAML (the same file passed to bitfount run_pod). Used to materialise the datasource and its schema exactly as the live pod would — required because the inference steps need a real datasource, model resource, and schema.
  • datasource_name: Name of the datasource to run against (must exist in the pod config).
  • project_id: Optional project ID for Hub completion reporting and model access gating. Defaults to a generated local run ID.