Conformal Prediction Demo using TensorFlow.js

Epsilon: 0.200
Starting...
CP Statistics
Error rate
Average pred set size

Notes

  1. This page demonstrates Conformal Predictors on the MNIST digit recognition problem. It is a pure JavaScript implementation, so all computation takes place in the browser (rather than on a server).

  2. The implementation uses a CNN implemented in Tensorflow JS to recognize the digits (based on the mnist-core) example.

  3. Upon starting, the demo downloads the MNIST image data and trains (partially) the CNN. A status line below the heading shows the progress of the training.

  4. The central area shows a sample of 100 test images. The background is green when the CP prediction is correct, red in case of error (i.e. the prediction set not containing the actual label). The small numbers in yellow at the bottom of each image represent the prediction set.

  5. The big slider allows to vary the CP significance level 'epsilon'. The p-values and the prediction sets are recomputed in real-time.
    One can observe how the error rate shown on the right follows the significance level, within statistical fluctuation. Also, one can see how the prediction sets become larger as the significance level get smaller.

  6. The buttons on the left allow the user to
    1. take a different sample from the test set.
    2. perform a few more steps in the training of the CNN.
    The latter can be used to demonstrate that the effect of the increased accuracy of the ML method underlying the CP.
    One can see that, by performing more training and therefore increasing (hopefully) the accuracy of the CNN, the prediction sets become smaller. The average prediction set size (where the average is over the sets with at least one element) is shown in the left panel.
  7. Technical details: the MNIST dataset used in this demo has 65,000 images. 4/6 are used as training set, 1/6 for calibration and 1/6 as test. In fact, because of some limitation in the WebGL backend, the calibration set was restricted to 2,000 examples.