Skip to main content

hashing

Per-step task_hash derivation — a Merkle hash over a step's config chain.

Every v9 cache table keys its rows on task_hash. That column used to hold generate_prefect_task_hash(pod, datasource) — the datasource identity alone, shared by every step of every task on that datasource — which was both a correctness bug and a reuse ceiling: editing a step's config left the hash unchanged, so the next run served rows computed under the old config, and one hash for the whole task could not express "the inference results are still valid, only the calculation config moved".

A step's hash here is instead derived from its own semantic config plus the hashes of the steps it consumes, so it changes exactly when its own config changes or when something upstream does. A config edit invalidates that step and its descendants while leaving expensive upstream work reusable.

Hash definition​

task_hash(step) = sha256_hex(canonical_json({
    "scheme":        HASH_SCHEME_VERSION,
    "datasource":    <leaf hash, or the datasource *type* — see below>,
    "task":          step.task,
    "version":       step.version,
    "config":        <config minus OPERATIONAL_CONFIG_FIELDS>,
    "resources":     <identity of each declared external resource>,
    "datastructure": <datastructure identity, or "" if the step ignores it>,
    "parents":       sorted({task_hash(dep) for dep in upstream step refs}),
}))[:32]   # [:32] matches the task_hash column width

The datasource component keeps two datasources on one pod out of each other's rows, and which form a step gets depends on the shape of its table:

  • A table keyed per entity — a file (model_inferences, the *_calculation tables) or a patient (ehr_data, ehr_patient_ids) — with no project column holds facts about that entity. Two datasources can only collide there on the same entity, and there they agree, so the component is the datasource's type and an entity present in several same-type datasources is stored once.
  • A project-keyed table gets the leaf hash — the eligibility trio, whose rows their consumers aggregate. Sharing those would merge two runs' outputs rather than deduplicate one entity's work.

registry.partition_is_entity_keyed decides from the ORM's declared primary key, so the rule cannot drift from the schema.

Parents are the step's FromRef / BackgroundRef inputs resolved to the referenced step's hash — a sorted, de-duplicated list of hashes only, never a {step name: hash} map. So neither input declaration order nor the names of upstream steps can move a hash: hashing names would make renaming a step a cache-wide event, and would stop two structurally identical flows from sharing partitions. The name->hash map a step needs at runtime is the separate, unhashed DAGStep.parent_task_hashes.

ContextRef inputs contribute nothing, but only because every provider is datasource-scoped and so already covered by the leaf hash; DATASOURCE_SCOPED_CONTEXT_PROVIDERS keeps that a checked premise.

Inputs that are not step config​

Two things determine a step's results without appearing in its config, and both arrive as parameters rather than being read here — which is what keeps this module a pure function of the parsed DAG (no I/O, no runtime data). That purity is what lets the interactive phase recompute a background step's hash and land on the value the background phase wrote under: both derive these from the same pod config and FlowSpec.

  • External resources. The EHR steps read a system named nowhere in their config — ehr_patient_lister has no config at all — and neither table's key saves them: ehr_patient_ids is keyed on the EHR's own patient IDs, and ehr_data on bitfount_patient_id, a hash of name + DOB that is deliberately identical across systems. Such a step declares task_hash_resources (see registry) and the caller supplies the matching identity in resource_identities; a declared resource with none raises rather than partitioning blind.
  • The datastructure. data_structure is a task-level YAML block that reaches a step as the datastructure runtime parameter, yet transform.image, select and assign all change the tensor a model sees. Only steps declaring that parameter fold it in (asked of registry.declares_runtime_param, so the signature is the single source of truth); the rest hash "", so retuning a transform re-partitions inference and its descendants but leaves the EHR steps alone.

What a step config may contain​

Config hashing is opt-out: every declared field enters the hash unless named in OPERATIONAL_CONFIG_FIELDS. tests/bitfount/flows/dag/test_hashing.py enforces both consequences across the whole step registry:

  • A field that varies between runs of the same task (an output path, a timestamp, a generated id) must be listed there, or it re-partitions the cache every run.
  • A config must not declare a set / frozenset. Pydantic serialises one in iteration order, which varies per process under string hash randomisation — a valid payload with a per-process digest, so the cache would never hit. Use a sorted list.

Every step, no exceptions​

Every DAG step is keyed by its own hash, the eligibility trio included. Two consequences for callers, because a step's hash is not its sibling's:

  • A step reading another step's table must go through the partition that step wrote. Most get this for free — the CacheAccessor handed down a DAG edge has the upstream hash baked into its filters. A step needing a store-level query an accessor cannot express reads DAGStep.parent_task_hashes[<its own input parameter name>], as patient_eligibility does.
  • trials_published_data_pointer publishes a {cache table: task_hash} map, so a reader crossing between two of a run's tables stays within that run's output.

Datasource/run-level tables outside the DAG (file_metadata, scan_metadata, runs, schema_versions) are untouched: the file-metadata runtime and the executor's run bookkeeping use the leaf hash directly.

Module​

Functions​

compute_step_task_hashes​

def compute_step_task_hashes(    steps: Sequence[DAGStep],    datasource_hash: str,    resource_identities: Mapping[str, str] | None = None,    datastructure_identity: str | None = None,    datasource_type_identity: str | None = None,) ‑> list[str]:

Compute each step's task_hash, in the order the steps are given.

The digests only; stamp_step_task_hashes is what a run uses, and it also assigns the parent routing tables. See _hash_steps for the arguments and the errors raised.

Returns One hash per step, positionally aligned with steps.

stamp_step_task_hashes​

def stamp_step_task_hashes(    dag: BackgroundDAG | InteractiveDAG,    datasource_hash: str,    resource_identities: Mapping[str, str] | None = None,    datastructure_identity: str | None = None,    datasource_type_identity: str | None = None,) ‑> dict[str, str]:

Compute and assign DAGStep.task_hash and .parent_task_hashes for dag.

Mutates the steps in place so each carries its own partition key: the executor injects it as the step's task_hash kwarg, and BackgroundResultsContext reads a background step's hash straight off the DAGStep it already holds — which is what keeps the two phases in agreement with no extra plumbing.

Each step also gets parent_task_hashes (see _resolve_parents), the same resolution the hashed payload used rather than a second one.

An interactive DAG's background_steps are hashed first and in the same pass, so a BackgroundRef's parent hash is the value the background phase computed. That holds only while both phases pass the same resource_identities and datastructure_identity; both derive them from the same pod config and FlowSpec, so they do.

Arguments

  • dag: A BackgroundDAG or InteractiveDAG. An InteractiveDAG's background_steps are hashed and stamped too.
  • resource_identities: Identity string per external resource kind, for steps declaring task_hash_resources.
  • datasource_hash: The datasource leaf hash — generate_prefect_task_hash(pod_name, datasource_name).
  • datastructure_identity: Canonical identity of the task's data_structure, for steps declaring a datastructure parameter.
  • datasource_type_identity: Identity of the datasource's type, for steps whose partition is keyed per file.

Returns Mapping of step name → assigned hash (interactive steps win on a name clash with a background step), for logging and assertions.