Skip to main content

metrics

Defines functions which compute metrics.

Module​

Functions​

sum_fpr_curve​

def sum_fpr_curve(    test_target: List[float],    test_pred: List[float],    test_var_to_sum: List[float],    test_var_margin: Optional[List[float]] = None,    granularity: int = 100,) ‑> Tuple[List[float], List[float]]:

Returns the false positive rate to sums of a given variable.

Returns this sum as a tuple of arrays (fpr, sums).

This is calculated by moving a threshold that is used for deciding to accept/reject a given entry and then summing the associated variable. This can be used, for example, to calculate the total profit that would be achieved according to different false positive rates.

Granularity is the number of regions to have that split up the threshold space.

Classes​

ClassificationMetric​

class ClassificationMetric(    func: Callable[[np.ndarray, np.ndarray], float], probabilities: bool,):

A classification metric used for assessing ML model performance.

Arguments

  • probabilities: Whether y_pred needs to be classes or probabilities.

Attributes

  • func: A function which computes the metric. Must take two arguments: y_true and y_pred and return a metric as a float.

Ancestors​

Variables​

  • static probabilities : bool

Metric​

class Metric(func: Callable[[np.ndarray, np.ndarray], float]):

A metric used for assessing ML model performance.

Attributes

  • func: A function which computes the metric. Must take two arguments: y_true and y_pred and return a metric as a float.

Subclasses​

Variables​

MetricCollection​

class MetricCollection(    metrics: Optional[MutableMapping[str, Metric]] = None,    problem: Optional[MetricsProblem] = None,):

Container class for metrics to calculate.

Arguments

  • metrics: A list of metrics to calculate.
  • problem: The problem type. If metrics are not specified, the problem type will be used to determine the metrics to calculate.

Attributes

  • metrics: A list of metrics to calculate.
  • problem: The problem type.
  • optimal_threshold: The optimal threshold to separate classes (only used for classification problems).
  • thresholds: The thresholds to separate classes (only used for classification problems).
  • threshold_metrics: The metrics for each threshold (only used for classification problems).

Raises

  • ValueError: If neither one of problem nor metrics is specified.

Static methods​

create_from_model​

def create_from_model(    model: Union[_BaseModel, DistributedModelProtocol],    metrics: Optional[MutableMapping[str, Metric]] = None,) ‑> MetricCollection:

Creates a MetricCollection object from a _BaseModel.

Arguments

  • model: A _BaseModel.
  • metrics: The metrics dictionary. Defaults to None.

Returns Instance of MetricCollection.

Methods​

compute​

def compute(    self,    test_target: np.ndarray,    test_preds: np.ndarray,    metric_to_optimise: str = 'F1',    threshold: Optional[float] = None,) ‑> Dict[str, float]:

Compute list of metrics and save results in self.results.

note

Thresholds do not apply to multiclass problems.

Arguments

  • test_target: A list of targets.
  • test_preds: A list of predictions.
  • metric_to_optimise: What metric to optimize in order to compute the optimal threshold. This will have no effect if there aren't any metrics to which a threshold is applied. Must be present in 'self.metrics'.
  • threshold: If this argument is provided, this threshold will be used instead of optimising the threshold as per 'optimise'

get_results_df​

def get_results_df(self) ‑> pandas.core.frame.DataFrame:

Calls get_results and returns results as a dataframe.

MetricsProblem​

class MetricsProblem(    value, names=None, *, module=None, qualname=None, type=None, start=1,):

Simple wrapper for different problem types for MetricCollection.

Ancestors​

Variables​

  • static BINARY_CLASSIFICATION
  • static MULTICLASS_CLASSIFICATION
  • static MULTILABEL_CLASSIFICATION
  • static REGRESSION
  • static SEGMENTATION