Skip to main content

dataframe

DataFrame conversion and manipulation utilities for steps.

Module​

Functions​

add_age_column​

def add_age_column(    df: pandas.core.frame.DataFrame,    dob_col: str = "Patient's Birth Date",    age_col: str = 'Age (yrs)',) ‑> pandas.core.frame.DataFrame:

Add an age column to df based on a date-of-birth column.

If age_col already exists and is fully populated the dataframe is returned unchanged. If the column exists but contains NA / None values, this function falls through and fills those gaps from dob_col, leaving any already-populated rows untouched.

Unparseable DOB values (e.g. de-identification placeholders like "XXXX") are coerced to NaT; age is computed only for rows with a valid DOB and left as NA for the rest.

If DOB parsing raises OverflowError (e.g. DOB set to Timestamp.min as a placeholder), a timedelta-based fallback is used.

Returns The same dataframe with the extra age column.

convert_ga_metrics_to_df​

def convert_ga_metrics_to_df(    ga_metrics: collections.abc.Mapping[str, typing.Any],    additional_pathology_prob_cols: collections.abc.Collection[str] | None = None,    additional_area_cols: collections.abc.Collection[str] | None = None,    additional_run_length_cols: collections.abc.Collection[str] | None = None,    metrics_type: type | None = None,) ‑> pandas.core.frame.DataFrame:

Convert a dict of GAMetrics objects into a dataframe.

Thin back-compat wrapper over convert_metrics_to_df; preserves the GA-specific exemplar-by-type selection between GAMetrics and GAMetricsWithFovea for expected_cols(), while always guarding record building on the base GAMetrics type. This matters when ga_metrics mixes plain GAMetrics and GAMetricsWithFovea values: the resolved exemplar type may narrow to GAMetricsWithFovea (to pick up the fovea columns), but a plain GAMetrics entry must still have its base fields recorded rather than being dropped entirely for failing an isinstance check against the narrower subclass.

Arguments

  • ga_metrics: Mapping of filename to GAMetrics instances.
  • additional_pathology_prob_cols: Additional pathology names to extract into max_<name>_probability columns.
  • additional_area_cols: Additional biomarker labels to extract into <label>_area columns.
  • additional_run_length_cols: Additional N-scan biomarker labels to extract into n_scan_run_<label> columns.
  • metrics_type: Expected type of GAMetrics instances (fallback when all None).

Returns DataFrame with each row representing GAMetrics for a single file.

convert_metrics_to_df​

def convert_metrics_to_df(    metrics: collections.abc.Mapping[str, typing.Any],    metrics_type: type[MetricsLike],    additional_pathology_prob_cols: collections.abc.Collection[str] | None = None,    additional_area_cols: collections.abc.Collection[str] | None = None,    additional_run_length_cols: collections.abc.Collection[str] | None = None,    record_isinstance_type: type | None = None,) ‑> pandas.core.frame.DataFrame:

Convert a mapping of filename to metrics objects into a wide DataFrame.

Works for any metrics dataclass that exposes to_record() and expected_cols() (e.g. GAMetrics, GAMetricsWithFovea, or the fluid-volume/CST/GCC metrics dataclasses). Values that are not instances of record_isinstance_type (defaulting to metrics_type; e.g. plain-string error reasons stored in place of a successful result) are treated as empty records rather than raising, so an error placeholder still produces a row with all metric columns unset.

Arguments

  • metrics: Mapping of filename to metrics instances (or non-metrics error values, e.g. strings, which produce an empty record).
  • metrics_type: The metrics dataclass type these records should conform to; used to look up expected_cols(), and (unless record_isinstance_type is given) also for the record-building isinstance guard.
  • additional_pathology_prob_cols: Additional pathology names to extract into max_<name>_probability columns, for metrics types (such as GAMetrics) whose to_record() supports this argument.
  • additional_area_cols: Additional biomarker labels to extract into <label>_area columns, for metrics types (such as GAMetrics) whose to_record() supports this argument.
  • additional_run_length_cols: Additional N-scan biomarker labels to extract into n_scan_run_<label> columns, for metrics types (such as GAMetrics) whose to_record() supports this argument.
  • record_isinstance_type: The type used for the record-building isinstance guard, if different from metrics_type. Useful when metrics_type is a subclass (e.g. GAMetricsWithFovea) but values of the base class (e.g. GAMetrics) should still have their base fields recorded rather than being dropped entirely. Defaults to None, meaning metrics_type is used for both.

Returns DataFrame with one row per mapping entry, an ORIGINAL_FILENAME_METADATA_COLUMN column of the mapping keys, and any missing expected columns added (and logged) as empty columns.

convert_predictions_to_dataframe​

def convert_predictions_to_dataframe(predictions: Any) ‑> pandas.core.frame.DataFrame:

Convert PredictReturnType to DataFrame if necessary.

Arguments

  • predictions: The predictions from the model algorithm.

Returns The predictions as a DataFrame.

cst_metrics_from_cache​

def cst_metrics_from_cache(    accessor: Any,) ‑> dict[str, CSTMetrics | None]:

Rebuild a filename→CSTMetrics map from a cst_calculation cache.

Typed wrapper over the shared metrics_from_cache for the CST background-phase calc cache, mapping an uncomputable file (null metrics_json) to None.

Arguments

  • accessor: A CacheAccessor for the cst_calculation cache partition.

Returns A {file_id: CSTMetrics | None} map.

fluid_metrics_from_cache​

def fluid_metrics_from_cache(    accessor: Any,) ‑> dict[str, FluidVolumeMetrics | None]:

Rebuild a filename→FluidVolumeMetrics map from a fluid_calculation cache.

Typed wrapper over the shared metrics_from_cache for the fluid-volume background-phase calc cache, mapping an uncomputable file (null metrics_json) to None.

Arguments

  • accessor: A CacheAccessor for the fluid_calculation cache partition.

Returns A {file_id: FluidVolumeMetrics | None} map.

ga_metrics_from_cache​

def ga_metrics_from_cache(    accessor: Any,) ‑> dict[str, GAMetricsWithFovea | None]:

Rebuild a filename→GAMetricsWithFovea map from a ga_calculation cache.

For the v9 flows that run GA calculation in the background, a downstream interactive step receives the step's .cache accessor rather than the in-memory metrics. Thin GA-typed wrapper over the shared metrics_from_cache (which also handles schema-drift tolerance), mapping an uncomputable file (null metrics_json) to None.

Arguments

  • accessor: A CacheAccessor for the ga_calculation cache partition.

Returns A {file_id: GAMetricsWithFovea | None} map suitable for convert_ga_metrics_to_df.

gcc_metrics_from_cache​

def gcc_metrics_from_cache(    accessor: Any,) ‑> dict[str, GCCMetrics | None]:

Rebuild a filename→GCCMetrics map from a gcc_calculation cache.

Typed wrapper over the shared metrics_from_cache for the GCC background-phase calc cache, mapping an uncomputable file (null metrics_json) to None.

Arguments

  • accessor: A CacheAccessor for the gcc_calculation cache partition.

Returns A {file_id: GCCMetrics | None} map.

get_data_for_files​

def get_data_for_files(    datasource: Any,    filenames: list[str],    file_key_col: str | None = None,    use_cache: bool = True,) ‑> pandas.core.frame.DataFrame:

Retrieve data from a datasource for a given list of files.

The dataframe returned will be sorted to match the ordering of the files in filenames.

lesion_metrics_from_cache​

def lesion_metrics_from_cache(    accessor: Any,) ‑> dict[str, LesionMetrics | None]:

Rebuild a filename→LesionMetrics map from a lesion_calculation cache.

Typed wrapper over the shared metrics_from_cache for the area background-phase calc cache, mapping an uncomputable file (null metrics_json) to None.

Arguments

  • accessor: A CacheAccessor for the lesion_calculation cache partition.

Returns A {file_id: LesionMetrics | None} map.

metrics_from_cache​

def metrics_from_cache(    cache_df: pandas.core.frame.DataFrame,    metrics_type: type[~_MetricT],    keep: collections.abc.Collection[str] | None = None,) ‑> dict[str, typing.Optional[~_MetricT]]:

Reconstruct a {file_id: metrics instance | None} map from a cache frame.

The single shared reader for every cache-backed sink (GA via ga_metrics_from_cache, plus the biomarker_tabulation CST/GCC/fluid builders). Reads each row's schemaless metrics_json blob (via metrics_json_by_file_id) and rebuilds metrics_type from it directly — the blob's keys match the dataclass's fields one-to-one (GA persists asdict minus the raw probability arrays; CST/GCC/fluid persist their to_record()). A null blob (an uncomputable file) maps to None.

Reconstruction is TOLERANT: the calc caches are task_hash-scoped and upsert-only (never pruned), so a row written by an older app version can outlive a change to its dataclass. A blob whose keys no longer match the current fields is logged and mapped to None (treated as uncomputable) rather than raising TypeError and aborting the whole step over one file.

Arguments

  • cache_df: The metric step's cache partition frame (file_id + metrics_json columns).
  • metrics_type: The metrics dataclass to reconstruct (e.g. CSTMetrics).
  • keep: If given, only these file_ids are reconstructed; others are skipped entirely (not just dropped later), so a caller that knows its selection avoids reconstructing an unbounded never-pruned partition.

Returns A {file_id: metrics_type instance | None} map, insertion-ordered by the cache frame's rows.

metrics_json_by_file_id​

def metrics_json_by_file_id(    cache_df: pandas.core.frame.DataFrame,) ‑> dict[str, dict[str, typing.Any] | None]:

Map file_id → its metrics_json dict (or None) from a cache frame.

The single reader of a scan-metrics cache partition's schemaless metrics_json blob, shared by every cache-backed sink (ga_metrics_from_cache here and the biomarker_tabulation builders). A row whose metrics_json is null (an uncomputable file the calculation skipped/failed) maps to None; the caller decides what an absent metric becomes (a reconstructed dataclass, an all-NA row, ...). An empty frame — or one without a file_id column — yields an empty map.

Arguments

  • cache_df: A scan-metrics cache partition frame (file_id + metrics_json columns), e.g. from CacheAccessor.to_dataframe().

Returns A {file_id: metrics_json_dict | None} map, insertion-ordered by the frame's rows.

Classes​

MetricsLike​

class MetricsLike(*args, **kwargs):

Structural protocol for metrics dataclasses convertible to a DataFrame.

Any dataclass exposing a to_record() method and an expected_cols() classmethod satisfies this protocol, e.g. GAMetrics, GAMetricsWithFovea, or the fluid-volume/CST/GCC metrics dataclasses.

Ancestors​

Static methods​

expected_cols​

def expected_cols() ‑> list[str]:

Returns the expected columns for a dataframe.

Methods​

to_record​

def to_record(self) ‑> dict[str, typing.Any]:

Convert to a record format compatible with pd.DataFrame.from_records().