Skip to main content

metrics

Defines functions which compute metrics.

Module

Functions

sum_fpr_curve

def sum_fpr_curve(    test_target: List[float],    test_pred: List[float],    test_var_to_sum: List[float],    test_var_margin: Optional[List[float]] = None,    granularity: int = 100,)> Tuple[List[float], List[float]]:

Returns the false positive rate to sums of a given variable.

Returns this sum as a tuple of arrays (fpr, sums).

This is calculated by moving a threshold that is used for deciding to accept/reject a given entry and then summing the associated variable. This can be used, for example, to calculate the total profit that would be achieved according to different false positive rates.

Granularity is the number of regions to have that split up the threshold space.

Classes

ClassificationMetric

class ClassificationMetric(    func: Callable[[np.ndarray, np.ndarray], float], probabilities: bool,):

A classification metric used for assessing ML model performance.

Arguments

  • probabilities: Whether y_pred needs to be classes or probabilities.

Attributes

  • func: A function which computes the metric. Must take two arguments: y_true and y_pred and return a metric as a float.

Ancestors

Variables

  • static probabilities : bool

Metric

class Metric(func: Callable[[np.ndarray, np.ndarray], float]):

A metric used for assessing ML model performance.

Attributes

  • func: A function which computes the metric. Must take two arguments: y_true and y_pred and return a metric as a float.

Variables

MetricCollection

class MetricCollection(    metrics: Optional[MutableMapping[str, Metric]] = None,    problem: Optional[MetricsProblem] = None,):

Container class for metrics to calculate.

Arguments

  • metrics: A list of metrics to calculate.
  • problem: The problem type. If metrics are not specified, the problem type will be used to determine the metrics to calculate.

Attributes

  • metrics: A list of metrics to calculate.
  • problem: The problem type.
  • optimal_threshold: The optimal threshold to separate classes (only used for classification problems).
  • thresholds: The thresholds to separate classes (only used for classification problems).
  • threshold_metrics: The metrics for each threshold (only used for classification problems).

Raises

  • ValueError: If neither one of problem nor metrics is specified.

Static methods


create_from_model

def create_from_model(    model: Union[_BaseModel, DistributedModelProtocol],    metrics: Optional[MutableMapping[str, Metric]] = None,)> MetricCollection:

Creates a MetricCollection object from a _BaseModel.

Arguments

  • model: A _BaseModel.
  • metrics: The metrics dictionary. Defaults to None.

Returns Instance of MetricCollection.

Methods


compute

def compute(    self,    test_target: np.ndarray,    test_preds: np.ndarray,    metric_to_optimise: str = 'F1',    threshold: Optional[float] = None,)> Dict[str, float]:

Compute list of metrics and save results in self.results.

note

Thresholds do not apply to multiclass problems.

Arguments

  • test_target: A list of targets.
  • test_preds: A list of predictions.
  • metric_to_optimise: What metric to optimize in order to compute the optimal threshold. This will have no effect if there aren't any metrics to which a threshold is applied. Must be present in 'self.metrics'.
  • threshold: If this argument is provided, this threshold will be used instead of optimising the threshold as per 'optimise'

get_results_df

def get_results_df(self)> pandas.core.frame.DataFrame:

Calls get_results and returns results as a dataframe.

MetricsProblem

class MetricsProblem(    value, names=None, *, module=None, qualname=None, type=None, start=1,):

Simple wrapper for different problem types for MetricCollection.

Ancestors

Variables

  • static BINARY_CLASSIFICATION
  • static MULTICLASS_CLASSIFICATION
  • static MULTILABEL_CLASSIFICATION
  • static REGRESSION
  • static SEGMENTATION