0
DAYS
- Paper deadline: May 23rd May 30th (extended)
- Author notification: (paper) July 26th; (abstract) July 30th
- Poster submission (extended abstract up to 2 pages): July 30th
- Camera-ready: August 10th
Prof. Shen-Shyang Ho (Rowan University, USA)
Prof. Shen-Shyang Ho
Department of Computer Science at Rowan University, New Jersey, USA
Anomaly Detection in Data Streams using Martingales: New Research Issues and Results
Many real-world anomaly detection applications in dynamic environments require real-time detection of anomalies in a data streaming setting. Using the test of exchangeability based on martingale has shown to be an attractive approach with highly desirable characteristics in terms of efficiency and effectiveness in a data streaming setting.
In this talk, I will describe our recent work on how the detection delay time can be reduced by incorporating model prediction into the martingale test framework and demonstrate its feasibility using flight trajectory data. Moreover, I will present some preliminary results on the use of martingale test to detect anomalies in evolving graphs and how its output can be used to explain the detected anomalies.
Shen-Shyang Ho is an associate professor in the Department of Computer Science at Rowan University, New Jersey, USA. Previously, he was affiliated to Nanyang Technological University (Singapore), University of Maryland, California Institute of Technology, and NASA Jet Propulsion Laboratory. He received his Ph.D. in Computer Science from George Mason University in 2007 and his Bachelor (Honors) in Science (Mathematics and Computational Science) from the National University of Singapore in 1999.
His research has been funded by NSF, NASA, Rolls-Royce, BMW, National Research Foundation (Singapore), Ministry of Education (Singapore). His current research includes anomaly detection, reinforcement learning, federated learning, continual learning and their application to real-world problems.
General Paper
"Calibrating multi-class models"
Ulf Johansson, Tuwe Löfström and Henrik Boström
Student Paper
"Shapley-value based inductive conformal prediction"
William Lopez Jaramillo and Evgueni Smirnov
Welcome speech
Tutorial Session (1): "Conformal Prediction"
Henrik Linusson (Home Page)
Chair: Lars Carlsson
How good is your prediction? In risk-sensitive applications, it is crucial to be able to assess the quality of a prediction, however, traditional classification and regression models don't provide their users with any information regarding prediction trustworthiness. In contrast, conformal classification and regression models associate each of their multi-valued predictions with a measure of statistically valid confidence, and let their users specify a maximal threshold of the model's error rate - the price to be paid is that predictions made with a higher confidence cover a larger area of the possible output space.
This tutorial aims to provide its attendees with the knowledge necessary to implement conformal prediction in their daily data science work, be it research or practice oriented, as well as highlight current research topics on the subject.
Tea/Coffee Break
Tutorial Session (2): "Venn Predictors"
Ulf Johansson (Home Page)
Chair: Lars Carlsson
This tutorial first covers the basics of Venn predictors, including the special case of Venn-Abers predictors. After that, we consider Venn predictors in the context of trustworthiness. How to make decisions and recommendations from AI systems trustworthy is currently a key question for uptake and acceptance, often determining the success of such systems. Trustworthiness is also the ultimate goal of both Explainable AI (XAI) and Fairness, Accountability and Transparency (FAT).
Lunch Break
Invited talk
"Anomaly Detection in Data Streams using Martingales: New Research Issues and Results"
Shen-Shyang Ho (Department of Computer Science at Rowan University, New Jersey, USA)
Chair: Vladimir Vovk
Workshop: "Federated Machine Learning"
Ola Spjuth and Andreas Hellander (Scaleout Systems)
(Ola's Home Page)
(Andreas's Home Page)
Chair: Giovanni Cherubin
The first part of this workshop gives an introduction to federated machine learning (FedML) where multiple participants collaborate to develop machine learning models without needing to directly share data (that could be sensitive or confidential) with each other. The second part is a hands-on workshop on how to set up, run and deploy a federated learning project using the FEDn open source solution.
Paper presentation session (1)
Chair: Alex Gammerman
"Approximation to object conditional validity with inductive conformal predictors"
Anthony Bellotti
"Mondrian conformal predictive distributions"
Henrik Boström, Ulf Johansson and Tuwe Löfström
"A lower bound for a prediction algorithm under the Kullback-Leibler game"
Raisa Dzhamtyrova and Yuri Kalnishkan
"Shapley-value based inductive conformal prediction"
William Lopez Jaramillo and Evgueni Smirnov
Tea/Coffee Break
Paper presentation session (2)
Chair: Evgueni Smirnov
"Conformal testing in a binary model situation"
Vladimir Vovk
"Synergy conformal prediction"
Niharika Gauraha and Ola Spjuth
"Calibrating multi-class models"
Ulf Johansson, Tuwe Löfström and Henrik Boström
Lunch Break
Paper presentation session (3)
Chair: Henrik Boström
"Impact of model-agnostic nonconformity functions on efficiency of conformal classifiers: an extensive study"
Marharyta Aleksandrova and Oleg Chertov
"Using inductive conformal martingales for addressing concept drift in data stream classification"
Charalambos Eliades and Harris Papadopoulos
"Retrain or not retrain: conformal test martingales for change-point detection"
Vladimir Vovk, Ivan Petej, Ilia Nouretdinov, Ernst Ahlberg, Lars Carlsson and Alex Gammerman
"Conformal uncertainty sets for robust optimization"
Chancellor Johnstone and Bruce Cox
Paper presentation session (4)
Chair: Ulf Johansson
"Class-wise confidence for debt prediction in real estate management: discussion and lessons learned from an application"
Soundouss Messoudi, Sebastien Destercke and Sylvain Rousseau
"Evaluation of updating strategies for conformal predictive systems in the presence of extreme events"
Hugo Werner, Lars Carlsson, Ernst Ahlberg and Henrik Boström
"Transformer-based conformal predictors for paraphrase detection"
Patrizio Giovannotti and Alex Gammerman
"A non-conformity approach towards post-prostatectomy metastasis estimation using a multicentre prostate cancer database"
Christos Chatzichristos, Jose-Felipe Golib-Dzib, Andries Clinckaert, Wouter Everaerts, Maarten De Vos and Martine Lewi
Tea/Coffee Break
Poster session
Chair: Zhiyuan Luo
"Confidence machine learning for cutting tool life prediction"
Nishant Wilson, Steve Barwick, Vince Booker, Tom Mildenhall, Laura Still, Yan Wang and Khuong An Nguyen
"Protected probabilistic classification"
Vladimir Vovk, Ivan Petej and Alex Gammerman
"Conformal changepoint detection in continuous model situations"
Ilia Nouretdinov, Vladimir Vovk and Alex Gammerman
"Fast conformal classification using influence functions"
Umang Bhatt, Adrian Weller and Giovanni Cherubin
Lunch Break
Tutorial Session (3): "Conformal Prediction in Orange"
Tomaž Hočevar and Blaž Zupan, Faculty of Computer and Information Science, University of Ljubljana
(Tomaž's Home Page)
(Blaž's Home Page)
Chair: Lars Carlsson
Conformal prediction is a machine learning approach to report on the reliability of predictive models when applied to new cases. Machine learning techniques are gaining in complexity, and assessing their reliability may be an essential part of explaining the inner workings of predictive models. For practical purposes and dissemination of conformal prediction techniques, we must include these within easily accessible toolboxes. In machine learning, a significant subset of such toolboxes is those that use workflows and visual programming. Here, we report on an example of such a toolbox, Python implementation of conformal prediction library, and our initial efforts and ideas to democratize conformal prediction.
[Download PDF] [Download "conformal.ows"]
Closing address
The main purpose of conformal prediction is to complement predictions delivered by various algorithms of Machine Learning with provably valid measures of their accuracy and reliability under the assumption that the observations are independent and identically distributed. It was originally developed in the late 1990s and early 2000s but has become more popular and further developed in important directions in recent years.
Conformal prediction is a universal tool in several senses; in particular, it can be used in combination with any known machine-learning algorithm, such as SVM, Neural Networks, Ridge Regression, etc. It has been applied to a variety of problems from diagnostics of depression to drug discovery to the behaviour of bots.
A sister method of Venn prediction was developed at the same time as conformal prediction and is used for probabilistic prediction in the context of classification. Among recent developments are adaptations of conformal and Venn predictors to probabilistic prediction in the context of regression.
The COPA series of workshops is a home for work in both conformal and Venn prediction, as reflected in its full name “Conformal and Probabilistic Prediction with Applications”. The aim of this symposium is to serve as a forum for the presentation of new and ongoing work and the exchange of ideas between researchers on any aspect of Conformal and Probabilistic Prediction and their applications to interesting problems in any field.
Topics of the symposium include, but are not limited to:
Authors are invited to submit original, English-language research contributions or experience reports. Papers should be no longer than 20 pages formatted according to the well-known JMLR (Journal of Machine Learning Research) style. The LaTeX package for the style is available here.
Work-in-progress and extended abstract (up to 2 pages) are welcome for poster submission, and will be included in the conference's proceeding. The submission should be formatted according to the well-known JMLR (Journal of Machine Learning Research) style.
All aspects of the submission and notification process will be handled online via the EasyChair Conference System at:
There will be two Alexey Chervonenkis awards for the Best Paper and Best Student Paper, presented at the conference.
Researchers interested in Conformal Prediction may be interested in joining our online discussion group. Future announcements and related materials will be published regularly.
All accepted papers will be presented at the conference and published by the Proceedings of Machine Learning Research (PMLR). Volume 152.
Please make sure to use the correct style file; it should be already installed on your computer, or will be installed "on the fly". The camera-ready papers should follow the style of the Proceedings section of JMLR rather than JMLR itself.
In the end you should submit two files: (1) file containing your paper in the pdf format; (2) copyright form (http://proceedings.mlr.press/pmlr-license-agreement.pdf). Please upload the final version of your paper in pdf via EasyChair and email the signed copyright form to zhiyuan.luo@rhul.ac.uk
The beginning of your file will look like:
\documentclass[wcp]{jmlr} \usepackage{amsmath,amssymb,graphicx,url} \jmlrvolume{152} \jmlryear{2021} \jmlrworkshop{Conformal and Probabilistic Prediction and Applications} \jmlrproceedings{PMLR}{Proceedings of Machine Learning Research} \title{Nonparametric predictive distributions based on conformal prediction} \author{\Name{John Smith}\Email{j.smith@gmail.com}\\ \addr{Royal Holloway, University of London, Egham, Surrey, UK}\\ \Name{Minge Shen}\Email{m.shen@gmail.com}\\ \addr{Rutgers University, New Brunswick, NJ, USA}} \editor{Lars Carlsson, Zhiyuan Luo, Giovanni Cherubin and Khuong An Nguyen} \begin{abstract} This paper applies conformal prediction to derive predictive distributions that are valid under a nonparametric assumption. \end{abstract} \begin{keywords} Conformal prediction, predictive distributions, regression. \end{keywords}
Notice the presence of the command
\jmlrproceedings{PMLR}{Proceedings of Machine Learning Research}
Please let us know if you encounter any problems following these instructions.
For further information (which, however, can be confusing), see http://jmlr.org/proceedings/faq.html (under the heading "What is the Style File for the Proceedings?"); COPA 2021 uses the one column style file. The style file is available from the CTAN (Comprehensive TeX Archive Network) web site at http://ctan.org/tex-archive/macros/latex/contrib/jmlr you can go inside the directory called "sample_papers" and emulate the files jmlr_sample.tex and jmlr_sample.bib (the latter is only needed if you use bibtex).
A lot of useful advice on the JMLR style can be found at http://jmlr.org/format/format.html (however, please make sure to use the Proceedings style, as described above, rather than the main journal style).
To ensure the reproducibility of your results, and to help our community to flourish further, we encourage (but not require) that you share the code and data used for your experiments (if applicable). As a guideline, you may refer to the following link: https://github.com/paperswithcode/releasing-research-code.
Royal Holloway, University of London
Egham, United Kingdom