---
title: "Multiclass outcomes"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Multiclass outcomes}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 6,
  fig.height = 4,
  dpi = 110
)
```

## What "multiclass" means here

A multiclass task predicts one of three or more classes that carry no order.
`uniml` treats it as a separate task type rather than as a binary problem with a
positive class bolted on, because almost every binary report is built around a
single event: a threshold, a positive class, a calibration curve, a decision
curve. None of those survives contact with four unordered classes without a
statement of convention.

What the package fixes for you is the convention:

| Quantity | Definition |
| --- | --- |
| `auc` | Hand and Till multiclass AUC: the mean over ordered class pairs of the AUC on class `i`'s own score column (`mlr3measures::mauc_au1u`) |
| `auc_ovr` | Macro one-vs-rest AUC: each class against the rest, then the unweighted mean; equals `mean(per_class$auc)` |
| `micro_auc` | Micro-average AUC: all (patient, class) pairs pooled into one binary problem |
| `brier` | Multi-class Brier score: mean over rows of the summed squared class errors |
| `logloss` | Mean negative log probability of the observed class |
| `sensitivity`, `specificity`, `ppv`, `npv`, `f1` | Macro averages of the one-vs-rest values |
| `micro_specificity` | Specificity pooled over the one-vs-rest cells |
| `balanced_accuracy` | Mean per-class recall, which equals the macro sensitivity |
| `kappa` | Cohen's kappa on the k-by-k table |

There is no micro precision, micro recall, or micro F1. For a single-label
multiclass outcome all three are identically the accuracy — the same number three
times — so the package reports accuracy once and says why. Micro AUC and micro
specificity are the two micro numbers that carry information of their own.

Every number is reproducible from the confusion matrix that comes with it, and
`tests/testthat/test-multiclass.R` pins them to `mlr3measures` and to a direct
recomputation.

## A worked example

`datasets::iris` is used here because it ships with R and has three classes; the
mechanics are the same for a clinical outcome with three risk groups.

```{r data}
library(uniml)

data <- iris
data$Species <- factor(data$Species)
table(data$Species)

spec <- uniml_spec(
  task_type = "multiclass",
  outcome = "Species"
)
spec <- set_learners(spec, c("cart", "rf"), tune = TRUE)
spec
```

Note there is no `positive`: a positive class only exists when there are two.

```{r task}
prepared <- as_uniml_data(data, spec)
levels(prepared$outcome)

task <- make_task(data, spec)
task$class_names
task$positive
```

## Train and score

```{r train}
learner <- train_model(
  task,
  mlr3::lrn("classif.rpart", predict_type = "prob")
)
prediction <- learner$predict(task)
head(prediction$prob, 3)
```

`calc_metrics()` recognises a prediction with more than two probability columns
and switches to the multiclass metric set:

```{r metrics}
metrics <- calc_metrics(prediction)
metrics[setdiff(names(metrics), c("confusion", "per_class"))]
```

The confusion matrix has the reference classes in rows and the predicted classes
in columns, and the per-class table turns it into one-vs-rest rates:

```{r confusion}
metrics$confusion
metrics$per_class
```

The macro sensitivity and the balanced accuracy are the same quantity computed
two ways, which is a useful check rather than a duplication:

```{r macro}
data.frame(
  sensitivity = metrics$sensitivity,
  balanced_accuracy = metrics$balanced_accuracy,
  accuracy = metrics$accuracy,
  kappa = metrics$kappa
)
```

## Does a model survive resampling?

Nothing here is special-cased for the workflow, so ordinary mlr3 resampling
applies; `calc_metrics()` scores the pooled out-of-fold predictions.

```{r cv}
resampling <- mlr3::rsmp("cv", folds = 5L)
resampling$instantiate(task)
cv <- mlr3::resample(task, learner, resampling, store_models = FALSE)

pooled <- as_uniml_multiclass(cv$prediction(), truth = cv$prediction()$truth)
cv_metrics <- calc_metrics(pooled)
round(cv_metrics$auc, 3)
round(cv_metrics$accuracy, 3)
```

Tuning works the same way as for a binary task, because the search space and the
resampling are mlr3 objects. `resolve_learners()` returns the package's learner
specifications, which carry their search spaces:

```{r tune}
learners <- resolve_learners(spec, task = task)
tuned <- tune_model(
  task,
  learners[["cart"]],
  resampling = mlr3::rsmp("cv", folds = 3L),
  tuner = "random_search",
  terminator = mlr3tuning::trm("evals", n_evals = 4L),
  seed = 123
)
tuned$status
tuned$best_params
```

## Comparing models

`rank_models()` averages a composite score whose metric set is declared by the
task type, so a multiclass comparison is not ranked on two-class numbers. The
multiclass composite is AUC, accuracy, balanced accuracy, and macro F1.

```{r rank}
score_row <- function(model, prediction) {
  values <- calc_metrics(prediction)
  data.frame(
    model = model,
    set = "validation",
    auc = values$auc,
    accuracy = values$accuracy,
    balanced_accuracy = values$balanced_accuracy,
    f1 = values$f1,
    stringsAsFactors = FALSE
  )
}

forest <- train_model(task, mlr3::lrn("classif.ranger", predict_type = "prob"))
comparison <- rbind(
  score_row("cart", prediction),
  score_row("rf", forest$predict(task))
)
comparison
rank_models(comparison, task_type = "multiclass")
```

A plain metric data frame carries no task type, so `task_type` has to be passed;
a `uniml_result` remembers it for you. Ranking a binary result is unchanged.

## Figures

Discrimination is shown as one-vs-rest ROC curves, one per class. There is no
single ROC curve for a multiclass problem, and averaging them into one line
hides exactly the class you are worried about.

```{r roc, fig.cap = "One-vs-rest ROC curves."}
binary_prediction <- as_uniml_multiclass(prediction)
plot_roc(binary_prediction)
```

```{r confusion-plot, fig.cap = "Confusion matrix: reference classes in rows, predicted in columns."}
plot_confusion(binary_prediction)
```

## Calibration

A multiclass model outputs a whole vector of probabilities for every patient, so
"is the number right" has to be asked in two ways. Both are reported, because a
model can pass one and fail the other.

```{r calibration}
calibration <- multiclass_calibration(prediction$prob, prediction$truth)
calibration$summary
calibration$per_class
```

* **Top-label** calibration keeps the largest probability of each row as the
  confidence and asks whether patients with 0.8 confidence really belong to the
  predicted class 80% of the time. Its numbers are `ece` (expected calibration
  error) and `mce` (the worst bin).
* **Class-wise** calibration repeats a one-vs-rest comparison for every class and
  averages the per-class ECEs without weighting, so a rare class that is badly
  wrong cannot hide behind the common ones. That is `ece_classwise`.
* `level_bias` is `mean confidence - accuracy`. It is the closest thing to the
  binary intercept: the multinomial model has no single slope to report, and its
  analogue (temperature scaling) is a recalibration method rather than a
  diagnostic.

ECE and MCE are bin-dependent summaries, not proper scoring rules; rank models on
the Brier score or the log loss, and read ECE as "how wrong is the stated
probability when the model is wrong".

```{r calibration-plot, fig.cap = "Reliability of the multiclass predictions: top-label confidence (left) and one-vs-rest per class (right)."}
plot_calibration(binary_prediction, scope = "both")
```

The same numbers ride along with every other metric, so they can be tabulated and
bootstrapped without a second call:

```{r calibration-metrics}
metrics <- calc_metrics(prediction)
round(metrics$ece, 3)
round(metrics$ece_classwise, 3)
metrics$per_class[, c("class", "ece", "mce")]
```

## Recalibration

Measuring the miscalibration and fixing it are two different jobs. The method for
a multinomial model is temperature scaling: divide every logit by one fitted
scalar, `softmax(log(p) / T)`. `T > 1` softens an over-confident model, `T < 1`
sharpens an under-confident one.

```{r recalibrate}
recalibrated <- calibrate_probabilities(binary_prediction, folds = 5L)
recalibrated$parameters$fitted
recalibrated$metrics_before[, c("accuracy", "ece", "logloss")]
recalibrated$metrics_after[, c("accuracy", "ece", "logloss")]
```

Two properties are worth knowing before reading those numbers.

The division is monotone and is applied to every class in the same way, so
**temperature scaling cannot change which class the model ranks first**. The
confusion matrix, the accuracy, and every threshold-based metric come out
identical; only the probabilities attached to them move. A method that can only
make the model better at *reporting* and cannot make it worse at *deciding* is
worth having even though it has one parameter.

The fitted scalar is chosen to be no worse than leaving the model alone, and the
`calibrated` matrix is cross-fitted, so the before/after table is measured on
probabilities that were not used to fit the scalar. That honesty has a cost: on a
model that was already calibrated the cross-fitted log loss can come out a hair
worse, because each fold's scalar is a noisy estimate of 1.

```{r recalibrate-plot, fig.cap = "The same predictions after temperature scaling."}
plot_calibration(
  as_uniml_multiclass(recalibrated$calibrated, truth = binary_prediction$truth),
  scope = "top_label"
)
```

`calibrate_probabilities()` takes `new_probs` (and optionally `new_labels`) so a
scalar fitted on the validation set can be applied to the test set. The job never
crosses sets by accident: the scalar is a property of the model, and the before/
after table is reported on whichever set you hand it.

## What is deliberately missing

These are not gaps to be filled in later without a decision; each needs a
convention that a package should not choose silently.

* **Dirichlet calibration.** The stronger multiclass recalibration (Kull et al.
  2019) fits `softmax(W log p + b)`, which is `K^2 + K` parameters, can change the
  predicted class, and is not identifiable on the few hundred validation patients
  a clinical model usually has. Temperature scaling is the part of the idea that
  such a sample can support, and it is what `calibrate_probabilities()` does.
* **Decision-curve analysis.** Net benefit compares one event against no event.
  Run one class against the rest with `as_uniml_binary()` and `dca()` if that is
  the question you are asking.
* **NRI / IDI, DeLong, Hosmer-Lemeshow.** All three are defined for a single
  positive class. They are not implemented for `uniml_multiclass`.
* **`run_workflow()`.** The full workflow is binary-only for now: its prediction
  table, thresholds, calibrators, and post-model tests all assume one probability
  column. Calling it on a multiclass specification stops with a message pointing
  back at this vignette. The layers underneath it are not binary-only any more —
  the task, the metrics, the calibration, and the composite ranking are all
  declared by the task type — which is why the pieces above can be combined by
  hand.

## References

Hand DJ, Till RJ (2001). A simple generalisation of the area under the ROC curve
for multiple class classification problems. *Machine Learning*, 45, 171-186.
doi:10.1023/A:1010920819831

Sokolova M, Lapalme G (2009). A systematic analysis of performance measures for
classification tasks. *Information Processing & Management*, 45(4), 427-437.
doi:10.1016/j.ipm.2009.03.002

Cohen J (1960). A coefficient of agreement for nominal scales. *Educational and
Psychological Measurement*, 20(1), 37-46. doi:10.1177/001316446002000104

Brier GW (1950). Verification of forecasts expressed in terms of probability.
*Monthly Weather Review*, 78(1), 1-3.

Guo C, Pleiss G, Sun Y, Weinberger KQ (2017). On calibration of modern neural
networks. *Proceedings of the 34th International Conference on Machine Learning*,
70, 1321-1330.

Kull M, Perello-Nieto M, Kängsepp M, Silva Filho T, Song H, Flach P (2019). Beyond
temperature scaling: obtaining well-calibrated multi-class probabilities with
Dirichlet calibration. *Advances in Neural Information Processing Systems*, 32.

See `system.file("REFERENCES.md", package = "uniml")` for the full list.
