Work presented at CPRML 2015 for model selection using efficiency of conformal predictors. Work done in collaboration with Ritvik Jaiswal and Dr. Vineeth Balasubramanian from IIT Hyderabad

1. MODEL SELECTION
USING CONFORMAL
PREDICTORS
ABHAY GUPTA
INDIAN INSTITUTE OF TECHNOLOGY HYDERABAD

2. PROPOSED APPROACH
• Efficiency - means of selecting model parameters in classifiers
• Narrow conformal prediction regions desirable
• Model selection posed as an optimisation problem
• Objective to optimise the value of the S-criterion of efficiency
• k-Nearest Neighbour (k-NN) classifier used for validating
approach

3. PROPOSED METHODOLOGY
• k-Nearest Neighbour (k-NN) classifier used for validating idea
• Objective: Minimise S-criterion of efficiency
• The k which minimises the objective function in general gives high accuracy and
efficiency.
• S-criterion of efficiency is defined as:
• where are the p-values defined as follows:

4. PROPOSED METHODOLOGY
• Smaller values preferable for S-criterion
• Ensures smaller size for the prediction set →
• Intuition: For an incoming test point, we want most of the training points to have a
higher conformity score than test points
• Small values for the expression are desirable
• and are conformity scores for test and training points respectively

5. PROPOSED METHODOLOGY
• This gives a proxy for the S-criterion which is to be minimized
where n and m are the number of test and training points respectively.
• The conformity score for an incoming point for the k-NN classifier is defined as:

6. PROPOSED METHODOLOGY
• This leads us to the objective function:
where n, m and k are the number of test points, training points and the
number of nearest neighbours respectively.

7. EMPIRICAL STUDY
• Following datasets were used:
● All results averaged over 5 trials

8. EMPIRICAL STUDY → USPS DATASET
k vs OBJECTIVE FUNCTION
ρ = 0.8589
k vs ACCURACY
ρ = -0.9271

9. EMPIRICAL STUDY → USPS DATASET
(EFFICIENCY RESULTS)
k vs OBJECTIVE FUNCTION k vs PREDICTION SET SIZE
(80% CONFIDENCE)

10. EMPIRICAL STUDY → STANDARD WAVEFORM DATASET
k vs OBJECTIVE FUNCTION
ρ = -0.6501
k vs ACCURACY
ρ = 0.7989

11. EMPIRICAL STUDY → STANDARD WAVEFORM DATASET
(EFFICIENCY RESULTS)
k vs OBJECTIVE FUNCTION k vs PREDICTION SET SIZE
(80% CONFIDENCE)

12. CONCLUSIONS AND FUTURE/ONGOING WORK
• While validity is guaranteed, efficiency varies with classifier parameters
• Proposed approach shows promise – a baby step
– k vs Objective function (Validation Set) and k vs Accuracy (Test Set) are
negatively correlated
– As value of the objective function decreases, efficiency increases (expectedly)
• Future/Ongoing work
– What would other measures of efficiency lead to?
– Can we frame this as a convex/submodular/other objective function with
guaranteed performance bounds?