Ranked probability score

[topepo]

Ranked probability score (RPS) for ordinal outcomes is basically a modified Brier score. The cumulative probabilities are compared to an indicator for whether the class is less than or equal to the current one.

Technically, it is a probability metric and we can check that the truth factor is ordinal. Alternatively, we could make new metric tyes specifically for ordinal data.

Some references:

• Random Forests for Ordinal Response Data: Prediction and Variable Selection section 2
• Random Forest Estimation of the Ordered Choice Model section 5.2