interactive prior knowledge elicitation -...
TRANSCRIPT
Expert knowledge elicitation for interactive improvement of machine learning models
Helsinki Institute for Information Technology HIIT is a joint
research institute of Aalto University and the University of Helsinki
for basic and applied research on information technology.
Helsinki Institute for Information Technology HIIT is a joint research institute of Aalto University and
the University of Helsinki for basic and applied research on information technology.
Case study: Predicting citations of scientific articles
Evaluation and User Study
• Predict the relative number of citations a scientific article will get in
Artificial Intelligence domain, based on its title, abstract and keywords
• An expert user indicates whether the words in the article (features)
are relevant for the prediction task
Solving elicitation on a budget – User model
• Balance between querying additional input on either the most
promising relevant features (exploitation), or on the most uncertain
ones (exploration)
References[1] Soare, M., Ammad-ud-din, M., Kaski, S. (2016). Regression with n 1 by Expert Knowledge Elicitation. In the 15th IEEE International Conference on Machine Learning and Applications, pp. 734 – 739.[2] Afrabandpey, H, Peltola, T., Kaski, S. (2016). Regression Analysis in Small-n-Large-p Using Interactive Prior Elicitation of Pairwise Similarities. In FILM 2016, NIPS Workshop on Future of Interactive Learning Machines.[3] Micallef, L.*, Sundin, I.*, Marttinen, P.*, Ammad-ud-din, M., Peltola, T., Soare, M., Jacucci, G., Kaski, S. (2017). Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 547 – 552. (*equal contribution).[4] Daee, P., Peltola, T., Soare, M., Kaski, S. (2017). Knowledge Elicitation via Sequential Probabilistic Inference for High-Dimensional Prediction. In arXiv preprint arXiv:1612.03328.
• Reinforcement learning using linear bandit: Features with the highest
Upper Confidence Bounds are shown to the expert user for input
• At each iteration, the user model updates estimated relevance of
features based on expert’s previous inputs
Prediction model
• A standard linear regression model, expert
knowledge brought in by adjusting the prior
distributions of the model’s parameters
• Interaction improves predictions compared to the model that doesn’t
use any expert user input
• Our user model method provides significantly better elicitation than a
naive method with random selection
Iiris Sundin1, Pedram Daee1, Homayun Afrabandpey1, Tomi Peltola1, Marta Soare1,
Muhammad Ammad-ud-din1, Luana Micallef1, Pekka Marttinen1, Giulio Jacucci2, Samuel Kaski1
1 Helsinki Institute for Information Technology HIIT, Department of Computer Science, Aalto University
2 Helsinki Institute for Information Technology HIIT, Department of Computer Science, University of Helsinki
Interactive expert knowledge elicitation in small n, large p problems
Results
• User study with 23
participants
• Users indicated their
domain knowledge with
an interactive interface
• User input for 200
keywords
• Modeling and making predictions are challenging
when there are only few samples (n), but many
predictive features (p)
• Applications: Genomic medicine and other cases
where collecting more data is not feasible, or even
not possible
Interactive system brings an expert to the loop