This page demonstrates Conformal Predictors on the MNIST digit recognition problem.
It is a pure JavaScript implementation, so all computation takes place in the browser (rather than on a server).
The implementation uses a CNN implemented in Tensorflow JS to recognize the digits
(based on the mnist-core) example.
Upon starting, the demo downloads the MNIST image data and trains (partially) the CNN.
A status line below the heading shows the progress of the training.
The central area shows a sample of 100 test images.
The background is green when the CP prediction is correct, red in case of error
(i.e. the prediction set not containing the actual label).
The small numbers in yellow at the bottom of each image represent the
prediction set.
The big slider allows to vary the CP significance level 'epsilon'.
The p-values and the prediction sets are recomputed in real-time.
One can observe how the error rate shown on the right follows the significance level, within statistical fluctuation.
Also, one can see how the prediction sets become larger as the significance level get smaller.
The buttons on the left allow the user to
take a different sample from the test set.
perform a few more steps in the training of the CNN.
The latter can be used to demonstrate that the effect of the increased
accuracy of the ML method underlying the CP.
One can see that, by performing more training and therefore increasing (hopefully) the
accuracy of the CNN, the prediction sets become smaller. The average prediction set
size (where the average is over the sets with at least one element) is
shown in the left panel.
Technical details: the MNIST dataset used in this demo has 65,000 images.
4/6 are used as training set, 1/6 for calibration and 1/6 as test.
In fact, because of some limitation in the WebGL backend, the calibration set
was restricted to 2,000 examples.