A multiclass task predicts one of three or more classes that carry no
order. uniml treats it as a separate task type rather than
as a binary problem with a positive class bolted on, because almost
every binary report is built around a single event: a threshold, a
positive class, a calibration curve, a decision curve. None of those
survives contact with four unordered classes without a statement of
convention.
What the package fixes for you is the convention:
| Quantity | Definition |
|---|---|
auc |
Hand and Till multiclass AUC: the mean over ordered class pairs of
the AUC on class i’s own score column
(mlr3measures::mauc_au1u) |
auc_ovr |
Macro one-vs-rest AUC: each class against the rest, then the
unweighted mean; equals mean(per_class$auc) |
micro_auc |
Micro-average AUC: all (patient, class) pairs pooled into one binary problem |
brier |
Multi-class Brier score: mean over rows of the summed squared class errors |
logloss |
Mean negative log probability of the observed class |
sensitivity, specificity,
ppv, npv, f1 |
Macro averages of the one-vs-rest values |
micro_specificity |
Specificity pooled over the one-vs-rest cells |
balanced_accuracy |
Mean per-class recall, which equals the macro sensitivity |
kappa |
Cohen’s kappa on the k-by-k table |
There is no micro precision, micro recall, or micro F1. For a single-label multiclass outcome all three are identically the accuracy — the same number three times — so the package reports accuracy once and says why. Micro AUC and micro specificity are the two micro numbers that carry information of their own.
Every number is reproducible from the confusion matrix that comes
with it, and tests/testthat/test-multiclass.R pins them to
mlr3measures and to a direct recomputation.
datasets::iris is used here because it ships with R and
has three classes; the mechanics are the same for a clinical outcome
with three risk groups.
library(uniml)
data <- iris
data$Species <- factor(data$Species)
table(data$Species)
#>
#> setosa versicolor virginica
#> 50 50 50
spec <- uniml_spec(
task_type = "multiclass",
outcome = "Species"
)
spec <- set_learners(spec, c("cart", "rf"), tune = TRUE)
spec
#> <uniml_spec>
#> task_type: multiclass
#> outcome: Species
#> positive: none
#> leakage_policy: custom
#> in_cv: selection, balance, tuning
#> selection: lasso
#> verbose: 1, n_workers: 1Note there is no positive: a positive class only exists
when there are two.
learner <- train_model(
task,
mlr3::lrn("classif.rpart", predict_type = "prob")
)
prediction <- learner$predict(task)
head(prediction$prob, 3)
#> setosa versicolor virginica
#> [1,] 1 0 0
#> [2,] 1 0 0
#> [3,] 1 0 0calc_metrics() recognises a prediction with more than
two probability columns and switches to the multiclass metric set:
metrics <- calc_metrics(prediction)
metrics[setdiff(names(metrics), c("confusion", "per_class"))]
#> $auc
#> [1] 0.98
#>
#> $accuracy
#> [1] 0.96
#>
#> $balanced_accuracy
#> [1] 0.96
#>
#> $kappa
#> [1] 0.94
#>
#> $logloss
#> [1] 0.1431763
#>
#> $brier
#> [1] 0.07353731
#>
#> $sensitivity
#> [1] 0.96
#>
#> $specificity
#> [1] 0.98
#>
#> $ppv
#> [1] 0.9618894
#>
#> $npv
#> [1] 0.9805021
#>
#> $f1
#> [1] 0.9599359
#>
#> $auc_ovr
#> [1] 0.98
#>
#> $micro_auc
#> [1] 0.9905778
#>
#> $micro_specificity
#> [1] 0.98
#>
#> $ece
#> [1] 0
#>
#> $mce
#> [1] 0
#>
#> $ece_classwise
#> [1] 0
#>
#> $n_samples
#> [1] 150
#>
#> $n_classes
#> [1] 3The confusion matrix has the reference classes in rows and the predicted classes in columns, and the per-class table turns it into one-vs-rest rates:
metrics$confusion
#> predicted
#> observed setosa versicolor virginica
#> setosa 50 0 0
#> versicolor 0 49 1
#> virginica 0 5 45
metrics$per_class
#> class n tp fp fn tn sensitivity specificity ppv npv
#> 1 setosa 50 50 0 0 100 1.00 1.00 1.0000000 1.0000000
#> 2 versicolor 50 49 5 1 95 0.98 0.95 0.9074074 0.9895833
#> 3 virginica 50 45 1 5 99 0.90 0.99 0.9782609 0.9519231
#> f1 auc ece mce
#> 1 1.0000000 1.00 0 0
#> 2 0.9423077 0.97 0 0
#> 3 0.9375000 0.97 0 0The macro sensitivity and the balanced accuracy are the same quantity computed two ways, which is a useful check rather than a duplication:
Nothing here is special-cased for the workflow, so ordinary mlr3
resampling applies; calc_metrics() scores the pooled
out-of-fold predictions.
resampling <- mlr3::rsmp("cv", folds = 5L)
resampling$instantiate(task)
cv <- mlr3::resample(task, learner, resampling, store_models = FALSE)
pooled <- as_uniml_multiclass(cv$prediction(), truth = cv$prediction()$truth)
cv_metrics <- calc_metrics(pooled)
round(cv_metrics$auc, 3)
#> [1] 0.967
round(cv_metrics$accuracy, 3)
#> [1] 0.927Tuning works the same way as for a binary task, because the search
space and the resampling are mlr3 objects.
resolve_learners() returns the package’s learner
specifications, which carry their search spaces:
learners <- resolve_learners(spec, task = task)
tuned <- tune_model(
task,
learners[["cart"]],
resampling = mlr3::rsmp("cv", folds = 3L),
tuner = "random_search",
terminator = mlr3tuning::trm("evals", n_evals = 4L),
seed = 123
)
tuned$status
#> [1] "tuned"
tuned$best_params
#> $xval
#> [1] 10
#>
#> $minsplit
#> [1] 26
#>
#> $minbucket
#> [1] 14
#>
#> $cp
#> [1] 0.01286033
#>
#> $maxdepth
#> [1] 7rank_models() averages a composite score whose metric
set is declared by the task type, so a multiclass comparison is not
ranked on two-class numbers. The multiclass composite is AUC, accuracy,
balanced accuracy, and macro F1.
score_row <- function(model, prediction) {
values <- calc_metrics(prediction)
data.frame(
model = model,
set = "validation",
auc = values$auc,
accuracy = values$accuracy,
balanced_accuracy = values$balanced_accuracy,
f1 = values$f1,
stringsAsFactors = FALSE
)
}
forest <- train_model(task, mlr3::lrn("classif.ranger", predict_type = "prob"))
comparison <- rbind(
score_row("cart", prediction),
score_row("rf", forest$predict(task))
)
comparison
#> model set auc accuracy balanced_accuracy f1
#> 1 cart validation 0.9800 0.96 0.96 0.9599359
#> 2 rf validation 0.9992 0.98 0.98 0.9799980
rank_models(comparison, task_type = "multiclass")
#> model probability score rank metric set
#> 2 rf <NA> 0.9847995 1 composite validation
#> 1 cart <NA> 0.9649840 2 composite validationA plain metric data frame carries no task type, so
task_type has to be passed; a uniml_result
remembers it for you. Ranking a binary result is unchanged.
Discrimination is shown as one-vs-rest ROC curves, one per class. There is no single ROC curve for a multiclass problem, and averaging them into one line hides exactly the class you are worried about.
A multiclass model outputs a whole vector of probabilities for every patient, so “is the number right” has to be asked in two ways. Both are reported, because a model can pass one and fail the other.
calibration <- multiclass_calibration(prediction$prob, prediction$truth)
calibration$summary
#> n n_classes accuracy mean_confidence level_bias ece mce ece_classwise bins
#> 1 150 3 0.96 0.96 0 0 0 0 10
calibration$per_class
#> class n prevalence mean_predicted ece mce
#> 1 setosa 50 0.3333333 0.3333333 0 0
#> 2 versicolor 50 0.3333333 0.3333333 0 0
#> 3 virginica 50 0.3333333 0.3333333 0 0ece (expected calibration error) and
mce (the worst bin).ece_classwise.level_bias is mean confidence - accuracy.
It is the closest thing to the binary intercept: the multinomial model
has no single slope to report, and its analogue (temperature scaling) is
a recalibration method rather than a diagnostic.ECE and MCE are bin-dependent summaries, not proper scoring rules; rank models on the Brier score or the log loss, and read ECE as “how wrong is the stated probability when the model is wrong”.
plot_calibration(binary_prediction, scope = "both")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?The same numbers ride along with every other metric, so they can be tabulated and bootstrapped without a second call:
Measuring the miscalibration and fixing it are two different jobs.
The method for a multinomial model is temperature scaling: divide every
logit by one fitted scalar, softmax(log(p) / T).
T > 1 softens an over-confident model,
T < 1 sharpens an under-confident one.
recalibrated <- calibrate_probabilities(binary_prediction, folds = 5L)
recalibrated$parameters$fitted
#> $type
#> [1] "temperature"
#>
#> $temperature
#> [1] 1
#>
#> $nll_before
#> [1] 0.1431763
#>
#> $nll_after
#> [1] 0.1431763
#>
#> $converged
#> [1] TRUE
recalibrated$metrics_before[, c("accuracy", "ece", "logloss")]
#> accuracy ece logloss
#> 1 0.96 0 0.1431763
recalibrated$metrics_after[, c("accuracy", "ece", "logloss")]
#> accuracy ece logloss
#> 1 0.96 6.67821e-05 0.1466176Two properties are worth knowing before reading those numbers.
The division is monotone and is applied to every class in the same way, so temperature scaling cannot change which class the model ranks first. The confusion matrix, the accuracy, and every threshold-based metric come out identical; only the probabilities attached to them move. A method that can only make the model better at reporting and cannot make it worse at deciding is worth having even though it has one parameter.
The fitted scalar is chosen to be no worse than leaving the model
alone, and the calibrated matrix is cross-fitted, so the
before/after table is measured on probabilities that were not used to
fit the scalar. That honesty has a cost: on a model that was already
calibrated the cross-fitted log loss can come out a hair worse, because
each fold’s scalar is a noisy estimate of 1.
plot_calibration(
as_uniml_multiclass(recalibrated$calibrated, truth = binary_prediction$truth),
scope = "top_label"
)
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?calibrate_probabilities() takes new_probs
(and optionally new_labels) so a scalar fitted on the
validation set can be applied to the test set. The job never crosses
sets by accident: the scalar is a property of the model, and the before/
after table is reported on whichever set you hand it.
These are not gaps to be filled in later without a decision; each needs a convention that a package should not choose silently.
softmax(W log p + b), which is
K^2 + K parameters, can change the predicted class, and is
not identifiable on the few hundred validation patients a clinical model
usually has. Temperature scaling is the part of the idea that such a
sample can support, and it is what
calibrate_probabilities() does.as_uniml_binary() and dca() if that is the
question you are asking.uniml_multiclass.run_workflow(). The full workflow is
binary-only for now: its prediction table, thresholds, calibrators, and
post-model tests all assume one probability column. Calling it on a
multiclass specification stops with a message pointing back at this
vignette. The layers underneath it are not binary-only any more — the
task, the metrics, the calibration, and the composite ranking are all
declared by the task type — which is why the pieces above can be
combined by hand.Hand DJ, Till RJ (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 45, 171-186. doi:10.1023/A:1010920819831
Sokolova M, Lapalme G (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427-437. doi:10.1016/j.ipm.2009.03.002
Cohen J (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. doi:10.1177/001316446002000104
Brier GW (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3.
Guo C, Pleiss G, Sun Y, Weinberger KQ (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 70, 1321-1330.
Kull M, Perello-Nieto M, Kängsepp M, Silva Filho T, Song H, Flach P (2019). Beyond temperature scaling: obtaining well-calibrated multi-class probabilities with Dirichlet calibration. Advances in Neural Information Processing Systems, 32.
See system.file("REFERENCES.md", package = "uniml") for
the full list.