Skip to main content

directly_written_records

Cache records that must be migrated to a fixed version whenever the cache opens.

Most cache records reach their target version because a DAG step declares a cache_table/cache_table_version binding and flows/dag/setup pins it before the run. These records are the exception, for either of two reasons.

  • No step binding names them. They are written by direct store calls and read directly, so no CacheAccessor and no binding covers them — e.g. trials_published_data_pointer, flipped by patient_eligibility via set_published_pointer and read by the patient-data API.
  • A step binding names them, but a reader-only process reads a column above v1. A step's binding is only applied on a pod that runs the DAG, so it does nothing for a process that only reads — e.g. patient_level_eligibility, whose ehr_retrieved_at column (v2) the patient-data API reads per patient and whose partitions it uses to scope the unpartitioned patients listing, and model_inferences, whose v2 source_file_hash the API reaches through the full-row select(ModelInference) behind patient_api/scans/segmentations.py. Pinned here at the same version the step binds, so the two cannot disagree.

ensure_types migrates only what something pins. A record named by neither a step binding nor this map therefore stays at whatever version created it, and the first read of a column added in a later version fails with no such column on any pre-existing cache — which is exactly what happens to a reader (the patient-data API) that opens a cache no publishing DAG has migrated on this pod. So both paths pin this map:

  • flows/dag/setup._pinned_type_versions merges it into a run's step-derived pins, so a publishing pod migrates these records at DAG setup.
  • runtimes/file_metadata.open_background_cache passes it to ensure_types on every open, so a reader opening the pod cache stands these records at their target version without depending on a DAG having run first.

Hardcoding trades the registry's discovery-time validation for an explicit list, so the validation moves to a test: tests/bitfount/flows/dag/test_setup.py::test_every_directly_written_record_exists_at_that_version resolves every entry against the record registry, so a typo or a version that does not exist fails as a test rather than as no such column on someone's cache.

An entry can carry a real migration cost onto the reader's open: model_inferences reaches v3 by building the file_id index, and that table holds one row per inference per file, so a cache still at v1 or v2 with rows pays an index build proportional to row count when the API first opens it. A publishing DAG would have paid the same cost, and the alternative is the no such column failure above, so the cost is accepted rather than avoided.