HUMAN-IN-THE-LOOP EXTRACTION OF INTERPRETABLE CONCEPTS IN DEEP LEARNING MODELS
Abstract : The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-loop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinementQuantitative experiments show that our active learning approach can accurately extract meaningful visual concepts
? To tackle the issue of interpretability in neural networks, many techniques have been proposed to help people understand model predictions. TCAV (Testing with Concept Activation Vectors) and the follow-up work ACE aim to understand what signals the model uses for predicting different image labels They generate a measure of importance of a visual concept (e.g. wheel, glass) for a prediction (e.g. predicted as a car) in a trained model. However, the concepts generated by automatic clustering methods may not match human concepts. A prototype system implementing our proposed human-in-theloop workflow, featuring scalable image patch exploration, visual cues and interactive filters for active learning and a rich set of model diagnostics and comparative analysis visualizations
? To solve this problem, we use deep embeddings as a representation of the image patches. As an image is passed as an input through a DNN model, the output after each hidden layer is an embedding in that latent space. These deep embeddings provide hints for the model to distinguish different images. Previous work shows that euclidean distance1 in the latent space is an effective perceptual similarity metric .
Model developers encounter different problems while diagnosing their model to make improvements
? To solve this problem, we use deep embeddings as a representation of the image patches. As an image is passed as an input through a DNN model, the output after each hidden layer is an embedding in that latent space. These deep embeddings provide hints for the model to distinguish different images. Previous work shows that euclidean distance1 in the latent space is an effective perceptual similarity metric
? Model developers encounter different problems while diagnosing their model to make improvements
? In this view, each image patch could be treated as a multivariate data sample, including variables like prediction accuracy and concept confidence scores for the existing concept extractors. A barchart is displayed for each of these variables. To help the user quickly identify an interesting target and generate new facts, the crossfilter view is also connected with the image patch view. Only the selected image patches in the crossfilter will be plotted in the image patch view. These concept filters can help the user quickly identify confident or confused image patches for different concepts. It is particularly useful when the user has trained multiple visual concepts and would like to study how the learned concepts correlate with each other.
? The lost cargo has two types of pixel annotations: the coarse ones including lost cargo (obstacle), road, and background; the fine ones for distinguishing specific lost cargo objects/obstacles in the images like boxes, balls, and so on. The coarse annotations are used by DenseNetFCN for training and prediction. To quantitatively evaluate our concept extraction model, we use the fine annotations as groundtruth visual concepts.
? The coarse annotations are used by DenseNetFCN for training and prediction. To quantitatively evaluate our concept extraction model, we use the fine annotations as groundtruth visual concepts. We pick a concept — dogs and trained the concept classifier for 4 iterations
|