Skip to main content

ingest

Read an enrichment file into stored records.

Resolution, hashing, and CSV parsing. The row -> fact convention itself lives in entries.py; this module only locates the file and applies that convention per row.

Module​

Functions​

file_content_hash​

def file_content_hash(path: Path) ‑> str:

The sha256 of a file's contents, read in chunks.

Arguments

  • path: The file to hash.

Returns The hex digest.

records_from_csv​

def records_from_csv(    path: Path,    source: EnrichmentSource,    *,    task_hash: str,    source_hash: str,    processed_at: datetime,) ‑> list[PatientEnrichmentRecord]:

Parse one enrichment CSV into stored records.

A file whose columns cannot be determined — no header to read, a parse failure, or a configured column absent from whatever header was found — raises EnrichmentSourceUnreadableError rather than degrading to []: the caller replaces a source's entire stored row set with what this returns, so conflating "unreadable" with "readable and genuinely empty" would delete good rows on nothing more than a bad read. A file that parses cleanly and has a valid header but zero data rows returns [] for real — that is the one case in which replacing stored rows with nothing is correct.

A configured optional column that is present but whose values do not resolve stays per-row lenient: that cell's field is left unset, because a single unreadable eye or date is genuinely "unstated" rather than an error. What is not silent is the systematic case. Unresolved values are counted and reported once per source per ingest by _log_unresolved, warning when no row in the file resolved — a site exporting eye as 1/2 instead of L/R would otherwise rewrite the entire source unlateralised, delete the previously good rows, and stop every side: study_eye criterion qualifying, while the step logged nothing but a row count.

A row whose patient does not resolve is skipped rather than stored, and counted the same warn-once way by _log_unresolved_identity. A fact attributed to nobody cannot decide a criterion, and there is no safe fallback identity to give it. Whether the ID is read from the source's own column or derived from its name and date-of-birth columns is EnrichmentSource's decision; this applies it per row.

Arguments

  • path: The resolved file to read.
  • source: The source's config.
  • task_hash: The step's own cache partition key.
  • source_hash: The file's content hash, stored on every row.
  • processed_at: The ingest timestamp.

Returns One record per fact per readable row; [] for a readable file with no data rows.

Raises

  • EnrichmentSourceUnreadableError: If the file's columns could not be determined — it could not be parsed, a required column (the identity column(s), and every value column) is absent from its header, or a configured optional column (laterality_column / measured_at_column / mrn_column) is absent from its header.

resolve_enrichment_file​

def resolve_enrichment_file(file: str) ‑> pathlib.Path:

Resolve a configured file name under the enrichment directory.

Arguments

  • file: The configured file name.

Returns The resolved absolute path.

Raises

  • ValueError: If the resolved path falls outside the enrichment directory, or does not name a file within it. Config names a file, never a location, so an escaping or directory-valued path is a configuration error rather than a read to attempt.

Classes​

EnrichmentSourceUnreadableError​

class EnrichmentSourceUnreadableError(*args, **kwargs):

A source file could not be interpreted at all.

Distinct from a readable file that happens to have zero data rows (a valid header with no rows below it): that case is genuinely empty and records_from_csv returns [] for it, which the caller may correctly treat as "this source now has no facts" and replace its stored rows with nothing. This error means the opposite — the file's columns could not be determined at all (no header to read, a parse failure, or a configured column absent from whatever header was found) — so nothing is known about what the source's current rows should be, and replacing the previously stored rows with [] would silently delete good data rather than reflect anything real about this file.