# cutpointr benchmarks

#### Christian Thiele

#### 2022-04-13

To offer a comparison to established solutions, cutpointr will be benchmarked against `optimal.cutpoints` from the OptimalCutpoints package, ThresholdROC and custom functions based on the ROCR and pROC packages. By generating data of different sizes the benchmarks will offer a comparison of the scalability of the different solutions.

Using `prediction` and `performance` from the ROCR package and `roc` from the pROC package, we can write functions for computing the cutpoint that maximizes the sum of sensitivity and specificity. pROC has a built-in function to optimize a few metrics:

    # Return cutpoint that maximizes the sum of sensitivity and specificiy
    # ROCR package
    rocr_sensspec <- function(x, class) {
        pred <- ROCR::prediction(x, class)
        perf <- ROCR::performance(pred, "sens", "spec")
        sens <- slot(perf, "y.values")[[1]]
        spec <- slot(perf, "x.values")[[1]]
        cut <- slot(perf, "alpha.values")[[1]]
        cut[which.max(sens + spec)]
    }

    # pROC package
    proc_sensspec <- function(x, class) {
        r <- pROC::roc(class, x, algorithm = 2, levels = c(0, 1), direction = "<")
        pROC::coords(r, "best", ret="threshold", transpose = FALSE)[1]
    }

The benchmarking will be carried out using the microbenchmark package and randomly generated data. The values of the `x` predictor variable are drawn from a normal distribution which leads to a lot more unique values than were encountered before in the `suicide` data. Accordingly, the search for an optimal cutpoint is much more demanding, if all possible cutpoints are evaluated.

Benchmarks are run for sample sizes of 100, 1000, 1e4, 1e5, 1e6, and 1e7. For low sample sizes cutpointr is slower than the other solutions. While this should be of low practical importance, cutpointr scales more favorably with increasing sample size. The speed disadvantage in small samples that leads to the lower limit of around 25ms is mainly due to the nesting of the original data and the results that makes the compact output of `cutpointr` possible. This observation is emphasized by the fact that `cutpointr::roc` is quite fast also in small samples. For sample sizes > 1e5 cutpointr is a little faster than the function based on ROCR and pROC. Both of these solutions are generally faster than OptimalCutpoints and ThresholdROC with the exception of small samples. OptimalCutpoints and ThresholdROC had to be excluded from benchmarks with more than 1e4 observations due to high memory requirements and/or excessive run times, rendering the use of these packages in larger samples impractical.

    # ROCR package
    rocr_roc <- function(x, class) {
        pred <- ROCR::prediction(x, class)
        perf <- ROCR::performance(pred, "sens", "spec")
        return(NULL)
    }

    # pROC package
    proc_roc <- function(x, class) {
        r <- pROC::roc(class, x, algorithm = 2, levels = c(0, 1), direction = "<")
        return(NULL)
    }

n | task | OptimalCutpoints | ROCR | ThresholdROC | cutpointr | pROC
--- | --- | --- | --- | --- | --- | ---
1e+02 | Cutpoint Estimation | 2.288702 | 1.812802 | 1.194301 | 4.5018015 | 0.662101
1e+03 | Cutpoint Estimation | 45.056801 | 2.176401 | 36.239852 | 4.8394010 | 0.981001
1e+04 | Cutpoint Estimation | 2538.612001 | 5.667101 | 2503.801251 | 8.5662515 | 4.031701
1e+05 | Cutpoint Estimation | NA | 43.118751 | NA | 45.3845010 | 37.150151
1e+06 | Cutpoint Estimation | NA | 607.023851 | NA | 465.0032010 | 583.095000
1e+07 | Cutpoint Estimation | NA | 7850.258700 | NA | 5467.3328010 | 7339.356101
1e+02 | ROC curve calculation | NA | 1.732651 | NA | 0.7973505 | 0.447701
1e+03 | ROC curve calculation | NA | 2.035852 | NA | 0.8593010 | 0.694802
1e+04 | ROC curve calculation | NA | 5.662151 | NA | 1.8781510 | 3.658050
1e+05 | ROC curve calculation | NA | 42.820852 | NA | 11.0992510 | 35.329301
1e+06 | ROC curve calculation | NA | 612.471901 | NA | 159.8100505 | 610.433700
1e+07 | ROC curve calculation | NA | 7806.385452 | NA | 2032.6935510 | 7081.897251