Skip to main content

apply

Apply preprocessing to cached raw inference frames.

A cache table stores each row's raw payload packed into a single dict column (the model_inferences cache is the first such table — its packed column is inferences_json — but the machinery here is not specific to it). The postprocessor machinery, however, operates on a wide DataFrame (one column per field). apply_preprocessing bridges the two: it unpacks the packed dict column into wide columns, runs the existing create_postprocessors/apply_postprocessors pipeline (fetching any columns_required datasource metadata first), then re-packs the result back into the same column so downstream consumers read the shape they always have. The packed column name is configurable via inferences_column.

Module​

Functions​

apply_inference_preprocessing​

def apply_inference_preprocessing(    df: pd.DataFrame,    configs: list[PreprocessingConfig],    *,    datasource: BaseSource,    file_id_column: str = 'file_id',) ‑> pandas.core.frame.DataFrame:

Apply a preprocessing pipeline to a cached frame, wiring datasource metadata.

Convenience for consuming steps: given the pipeline a step has selected (by interrogating the frame's shape — see bitfount.preprocessing.detectors), applies it with a datasource-backed fetcher for any transform that needs extra columns (e.g. Columns/Rows for fovea scaling).

Arguments

  • df: A cached-inference DataFrame from a CacheAccessor.
  • configs: The ordered preprocessing-step configs to apply.
  • datasource: Datasource used to fetch any columns_required metadata.
  • file_id_column: Name of the file-id column on df.

Returns The frame with its packed dict column preprocessed. Returns df unchanged when it is empty or configs is empty.

apply_preprocessing​

def apply_preprocessing(    df: pd.DataFrame,    configs: list[PreprocessingConfig],    *,    fetch_metadata: MetadataFetcher | None = None,    file_id_column: str = 'file_id',    inferences_column: str = 'inferences_json',) ‑> pandas.core.frame.DataFrame:

Apply configs to the packed dict column of df.

Arguments

  • df: A cached-inference DataFrame (e.g. from CacheAccessor) with a file-id column and a packed dict column.
  • configs: Ordered preprocessing-step configs (a pipeline from bitfount.preprocessing.pipelines).
  • fetch_metadata: Callback supplying any columns_required datasource columns (e.g. Columns/Rows for fovea scaling). Required only when configs declare columns_required.
  • file_id_column: Name of the file-id column on df.
  • inferences_column: Name of the packed dict column on df.

Returns A copy of df with inferences_column replaced by the preprocessed (re-packed) dict. Returns df unchanged when it is empty or configs is empty.