Skip to main content

functions

Scan filter functions.

Ports _LatestFilterStrategy.apply from bitfount.federated.algorithms.filtering_algorithm (v8) into a pure function usable by the v9 scan_filter step. This is an additive port — the v8 filtering_algorithm.py is not modified.

Module​

Functions​

filter_latest_scans​

def filter_latest_scans(    df: pd.DataFrame, *, date_column: str, id_columns: list[str], num_latest: int,) ‑> pandas.core.frame.DataFrame:

Filter a DataFrame to keep only the latest records per unique ID.

Ported from _LatestFilterStrategy.apply (v8 bitfount.federated.algorithms.filtering_algorithm). Combines id_columns into a single COMBINED_ID_COLUMN (joined with "-"), coerces date_column to datetime (unparseable values become NaT and sort last rather than raising), then keeps the first num_latest rows per ID after sorting by ID then date descending.

Unlike the ported v8 method (which mutates its input DataFrame in place), this function operates on a copy so it has no side effects on the caller's DataFrame.

Arguments

  • df: The input DataFrame to be filtered.
  • date_column: The name of the column containing date information.
  • id_columns: Names of the columns to combine into the grouping ID.
  • num_latest: The number of records to return for each ID.

Returns The filtered DataFrame containing only the latest records for each unique ID.

Raises

  • ValueError: If the filter arguments are invalid, or if the date column or any of the ID columns are not found in the DataFrame.