Package: OptHoldoutSize 0.1.0.3

Introduction | Estimation of optimal holdout size | Parameters $N$ and $k_1$ | Simulation of dataset | Computation of OHS using parametric method | Start with estimates of k2 at 10 values of n | Candidate values for n | Starting value for theta | Rough estimate for variance of k2 | Successively add new points | Resample k2(n), to avoid double-dipping effect | Transform to total cost | Save | Begin as for parametric approach | Transform estimated k2 to costs

Introduction | Setup | Assumptions throughout | General setup | Demonstration of algorithms | Parametric assumptions | Simulate data | Resample values d | Estimate a,b, and c from values nset and d | Estimate optimal holdout sizes using parametric method | Estimate optimal holdout sizes using semi-parametric (emulation) method | To draw plot: | These are our sets of training sizes and k2 estimates, which will be built up. | Go up to this many points | Estimate parameters | Find next suggested point, parametric assumptions satisfied | Find next suggested point, parametric assumptions not satisfied | New estimates of k2 | Update data | Sys.sleep(10) | To draw plot with np points (np can be set using the button) | Estimate parameters for parametric part of semi-parametric method | First panel | Second panel | Rates of convergence | Load datasets of 'next point' | Maximum number of training set sizes we will consider | Generate random 'next points' | Initialise matrices of records | ohs_array[n,i,j,k,l] is | using the first n training set sizes | the ith resample | using the jth version of k2 (j=1: pTRUE, j=2: pFALSE) | using the kth algorithm (k=1: parametric, k=2: semiparametric/emulation) | using the lth method of selecting next points (l=1: random, l=2: systematic) | Resamplings for parametric algorithm, random next point | Resamplings for semiparametric/emulation algorithm, random next point | Resamplings for parametric algorithm, nonrandom (systematic) next point | Resamplings for semiparametric/emulation algorithm, nonrandom (systematic) next point | Load data | Settings | Plot drawing function | Set up plot parameters | Number of estimates | Initialise | Plot medians | Ranges | Coarsening factor: coarsen OHS estimates to nearest value | Record lengths of rounded OHS numbers | Plot ranges | Add legend | Bottom panel setup | Root mean square errors | Draw lines | Extract matrices from aray | True OHS | True functions l

Introduction | Setup | Assumptions | General setup | Results | Emergence of optimal holdout size | Cost of true negative, false positive etc for computing total cost | Generate coefficients for underlying model and for predictions in h.o. set | Set ground truth coefficients, and the accuracy at baseline | Run simulation | If ninters>0, append coefficients for interaction terms to coefficients | Collate and save all data | Load data | Holdout set size in absolute terms | Colour intensities | Draw cost function | Cost values | Initialise plot for cost function | Plot Cost line | Standard Deviation Bands | Add legend | Draw k2 function | Evaluate k2(n)*(N-n) | Evaluate k2 | Scalinge factor (for plotting) | Initialise plot for k2 function | Line for estimated k2(n) | Add standard deviation of k2(n). | Line for estimated k2(n)*(N-n) | Add standard deviation for k2(n)*(N-n) | Draw k1*n function | Evaluate k1 | Line for estimated k1*n | Add standard deviation. Note this is scaled by (nobs-n_ho) | Loop through values of 'families' and 'interactions'