ner.utils.metrics module

ner.utils.metrics.compute_entity_f1(y_true: ndarray | Dict[str, List[Tuple[int]]], y_pred: ndarray | Dict[str, List[Tuple[int]]], average: str = 'weighted', token_idxs: List | ndarray | None = None) → float

Compute an average of entity-level F1 scores.

Parameters:

y_trueUnion[np.ndarray, Dict[str, List[Tuple[int]]]]

The true label IDs, or a dictionary of true named-entities mapped to a list of their respective spans (e.g., {"LOC": [(1, 1), (2, 5)]}).

y_predUnion[np.ndarray, Dict[str, List[Tuple[int]]]]

The predicted label IDs, or a dictionary of predicted named-entities mapped to a list of their respective spans (e.g., {"LOC": [(1, 1), (2, 5)]}).

average{“weighted”, “macro”}, default: “weighted”

The averaging method for aggregating entity-level F1 scores.

token_idxsOptional[Union[List, np.ndarray]], default: None

The token indices associated with y_true and y_test, when they are np.ndarray.

Returns:

float

An average of entity-level F1 scores.

ner.utils.metrics.compute_loss(loss_fn: Callable, preds: Tensor, labels: Tensor) → Tensor

Compute loss given a loss function, logits, and labels. Note: the logits and labels must be on the same device.

Parameters:

loss_fnCallable

The loss function (criterion).

predstorch.Tensor

The prediction IDs output by the model.

labelstorch.Tensor

The target label IDs.

Returns:

torch.Tensor

The computed loss, returned as a tensor on the same device as logits and labels.

ner.utils.metrics.compute_metrics(preds: Tensor, labels: Tensor, padding_mask: Tensor | None = None, labels_ignore_idx: int | None = None, other_ner_tag_idx: int | None = None, average: str = 'weighted')

Compute metrics given predictions and labels.

Parameters:

predstorch.Tensor

The prediction IDs output by the model.

labelstorch.Tensor

The target label IDs.

padding_maskOptional[torch.Tensor], default: None

The padding mask.

labels_ignore_idxOptional[int], default: None

If padding_mask is not provided, the padding_mask can be auto-inferred from labels_ignore_idx, if labels_ignore_idx is provided. Corresponds to PAD_NER_TAG in NER_ENCODING_MAP.

other_ner_tag_idxOptional[int]

The index of the “O” tag; if provided, the performance computation ignores the “O” tags. Corresponds to O in NER_ENCODING_MAP.

averagestr

The averaging method for aggregating entity-level F1 scores.

Returns:

Dict[str, float]

A dictionary of metrics, includes loss, precision, recall, accuracy, F1, and weighted-average of entity-level F1 scores.