Package: OptHoldoutSize 0.1.0.2

OptHoldoutSize: Estimation of Optimal Size for a Holdout Set for Updating a Predictive Score

Predictive scores must be updated with care, because actions taken on the basis of existing risk scores causes bias in risk estimates from the updated score. A holdout set is a straightforward way to manage this problem: a proportion of the population is 'held-out' from computation of the previous risk score. This package provides tools to estimate a size for this holdout set and associated errors. Comprehensive vignettes are included. Please see: Haidar-Wehbe S, Emerson SR, Aslett LJM, Liley J (2022) <doi:10.48550/arXiv.2202.06374> (in Annals of Applied Statistics) for details of methods.

Authors:Sami Haidar-Wehbe [aut], Sam Emerson [aut], Louis Aslett [aut], James Liley [cre, aut]

OptHoldoutSize_0.1.0.2.tar.gz
OptHoldoutSize_0.1.0.2.zip(r-4.7)OptHoldoutSize_0.1.0.2.zip(r-4.6)OptHoldoutSize_0.1.0.2.zip(r-4.5)
OptHoldoutSize_0.1.0.2.tgz(r-4.6-any)OptHoldoutSize_0.1.0.2.tgz(r-4.5-any)
OptHoldoutSize_0.1.0.2.tar.gz(r-4.7-any)OptHoldoutSize_0.1.0.2.tar.gz(r-4.6-any)
OptHoldoutSize_0.1.0.2.tgz(r-4.6-emscripten)
manual.pdf |manual.html
card.svg |card.png
OptHoldoutSize/json (API)
NEWS

# Install 'OptHoldoutSize' in R:
install.packages('OptHoldoutSize', repos = c('https://jamesliley.r-universe.dev', 'https://cloud.r-project.org'))
Datasets:

On CRAN:

Conda:

This package does not link to any Github/Gitlab/R-forge repository. No issue tracker or development information is available.

3.18 score 9 scripts 502 downloads 30 exports 8 dependencies

Last updated from:a1cf00a496. Checks:7 WARNING, 2 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-x86_64WARNING173
source / vignettesOK161
linux-release-x86_64WARNING165
macos-release-arm64WARNING146
macos-oldrel-arm64WARNING149
windows-develWARNING117
windows-releaseWARNING90
windows-oldrelWARNING98
wasm-releaseOK105

Exports:add_aspre_interactionsaspreaspre_k2ci_mincostci_ohscov_fnerror_ohs_emulationexp_imp_fngen_base_coefsgen_predsgen_respgrad_mincost_powerlawgrad_nstar_powerlawlogisticlogitmodel_predictmodel_trainmu_fnnext_noptimal_holdout_sizeoptimal_holdout_size_emulationoracle_predpowerlawpowersolvepowersolve_generalpowersolve_sepsi_fnsens10sim_random_aspresplit_data

Dependencies:latticeMatrixmatrixStatsmnormtmvtnormrangerRcppRcppEigen

ASPRE example

Rendered fromASPRE_example.Rmdusingknitr::rmarkdownon May 07 2026.

Last update: 2026-04-07
Started: 2022-02-09

Comparison of algorithms

Rendered fromcomparison_of_algorithms.Rmdusingknitr::rmarkdownon May 07 2026.

Last update: 2025-04-09
Started: 2022-02-09

Simulated example

Rendered fromsimulated_example.Rmdusingknitr::rmarkdownon May 07 2026.

Last update: 2025-04-09
Started: 2022-02-09

Readme and manuals

Help Manual

Help pageTopics
Add interaction terms corresponding to ASPRE modeladd_aspre_interactions
Computes ASPRE scoreaspre
Emulation-based OHS estimation for ASPREaspre_emulation
Cost estimating function in ASPRE simulationaspre_k2
Parametric-based OHS estimation for ASPREaspre_parametric
Data for example on asymptotic confidence interval for OHS.ci_cover_a_yn
Data for example on asymptotic confidence interval for min cost.ci_cover_cost_a_yn
Data for example on empirical confidence interval for min cost.ci_cover_cost_e_yn
Data for example on empirical confidence interval for OHS.ci_cover_e_yn
Confidence interval for minimum total cost, when estimated using parametric methodci_mincost
Confidence interval for optimal holdout size, when estimated using parametric methodci_ohs
Covariance function for Gaussian processcov_fn
Data for vignette showing general exampledata_example_simulation
Data for 'next point' demonstration vignette on algorithm comparison using emulation algorithmdata_nextpoint_em
Data for 'next point' demonstration vignette on algorithm comparison using parametric algorithmdata_nextpoint_par
Measure of error for emulation-based OHS emulationerror_ohs_emulation
Expected improvementexp_imp_fn
Coefficients for imperfect risk scoregen_base_coefs
Generate matrix of random observationsgen_preds
Generate responsegen_resp
Gradient of minimum cost (power law)grad_mincost_powerlaw
Gradient of optimal holdout size (power law)grad_nstar_powerlaw
Logisticlogistic
Logitlogit
Make predictionsmodel_predict
Train model (wrapper)model_train
Updating function for mean.mu_fn
Finds best value of n to sample nextnext_n
Data for vignette on algorithm comparisonohs_array
Data for vignette on algorithm comparisonohs_resample
Estimate optimal holdout size under parametric assumptionsoptimal_holdout_size
Estimate optimal holdout size under semi-parametric assumptionsoptimal_holdout_size_emulation
Generate responsesoracle_pred
Parameters of reported ASPRE datasetparams_aspre
Plot estimated cost functionplot.optholdoutsize
Plot estimated cost function using emulation (semiparametric)plot.optholdoutsize_emul
Power law functionpowerlaw
Fit power law curvepowersolve
General solver for power law curvepowersolve_general
Standard error matrix for learning curve parameters (power law)powersolve_se
Updating function for variance.psi_fn
Sensitivity at theshold quantile 10%sens10
Simulate random dataset similar to ASPRE training datasim_random_aspre
Split datasplit_data