[22] proposed a unified framework to distill the knowledge from clean labels and knowledge graph, which can be exploited to learn a better model from noisy labels. (2014). Deep learning has achieved excellent performance in var- ious computer vision tasks, but requires a lot of training examples with clean labels. Zhong, S., Tang, W., & Khoshgoftaar, T. M. (2005). ABSTRACT. Auxiliary image regularization for deep cnns with noisy labels. Oza, N. C. (2004) Aveboost2: Boosting for noisy data. Experiments with a new boosting algorithm. Freund, Y., Schapire, R. E., et al. (2018) Co-sampling: Training robust networks for extremely noisy supervision. Deep learning from noisy image labels with quality embedding. ACL materials are Copyright © 1963–2020 ACL; other materials are copyrighted by their respective copyright holders. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. (2000). The idea of using unbiasedestimators is well-knownin stochastic optimization[Nemirovskiet al., 2009], and regret bounds can be obtained for learning with noisy labels … The first series of noisy datasets we generated contain randomly dropped (ie. Patrini et al. 2019-ICLR_W - SOSELETO: A Unified Approach to Transfer Learning and Training with Noisy Labels. In, © Springer Nature Singapore Pte Ltd. 2020, Advances in Data and Information Sciences, http://proceedings.mlr.press/v37/menon15.html, https://doi.org/10.1007/s10994-013-5412-1, Department of Computer Science and Engineering, https://doi.org/10.1007/978-981-15-0694-9_38. Karmaker, A., & Kwek, S. (2006). Induction of decision trees. © 2020 Springer Nature Switzerland AG. Learning with Noisy Partial Labels by Simultaneously Leveraging Global and Local Consistencies. Part of Springer Nature. Zhu, X., Wu, X. Part of: Advances in Neural Information Processing Systems 26 (NIPS 2013) [Supplemental] Authors. Friedman, J., Hastie, T., Tibshirani, R., et al. Liu, H., & Zhang, S. (2012). Zhu, X., Wu, X., Chen, Q. For convenience, we assign 0 as the class label of samples belonging to background. Noise modelling and evaluating learning from examples. Abstract: The ability of learning from noisy labels is very useful in many visual recognition tasks, as a vast amount of data with noisy labels are relatively easy to obtain. Learning Adaptive Loss for Robust Learning with Noisy Labels. ICLR 2020 • Junnan Li • Richard Socher • Steven C. H. Hoi. (2014). The SpaceNet dataset contains a set of images, where for each image, there is a set of polygons in vector format, each representing the outline of a building. Tianrui Li. Bouveyron, C., & Girard, S. (2009). Learning with Noisy Class Labels for Instance Segmentation 5 corresponds to an image region rather than an image. Data quality and systems theory. Learning with Noisy Labels Nagarajan Natarajan, Ambuj Tewari, Inderjit Dhillon, Pradeep Ravikumar. This paper stud- ies the problem of learning with noisy labels for sentence-level sentiment classification. A study of the effect of different types of noise on the precision of supervised learning techniques. Learning with Noisy Labels for Sentence-level Sentiment Classification, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), https://www.aclweb.org/anthology/D19-1655, https://www.aclweb.org/anthology/D19-1655.pdf, Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License, Creative Commons Attribution 4.0 International License. Frénay, B., & Verleysen, M. (2014). Liu, T., & Tao, D. (2016). Unlike most existing methods relying on the posterior probability of a noisy classifier, we focus on the much richer spatial behavior of data in the latent representational space. y i is the class label of the sample x i and can be noisy. Vu, T. K., Tran, Q. L. (2018). Cantador, I., Dorronsoro, J. R. (2005). (2010). Noisy data is the main issue in classification. In PLL problem, the partial label set consists of exactly one ground-truth label and some other noisy labels. Here we focus on the recent progress on deep learning with noisy labels. Learning with noisy labels. Chaozhuo Li, For example, Li et al. At high sparsity (see next paragraph) and 40% and 70% label noise, CL outperforms Google’s top … Site last built on 14 December 2020 at 17:16 UTC with commit 201c4e35. Yan Yang, Traditionally, label noise has been treated as statistical outliers, and techniques such as importance re-weighting and bootstrapping have been proposed to alleviate the problem. Correcting noisy data. Deep learning from crowds. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. (2013). (2018). Robust loss functions: Defense mechanisms for deep architectures. Azadi, S., Feng, J., Jegelka, S., & Darrell, T. (2015). Loss factorization, weakly supervised learning and label noise robustness. (2017). Veit et al. (2015) Deep classifiers from image tags in the wild. Deep neural networks (DNNs) can fit (or even over-fit) the training data very well. In this section, we review studies that have addressed label noise in training deep learning models for medical image analysis. deleted) buildings. To re (label), or not to re (label). Nettleton, D. F., Orriols-Puig, A., & Fornells, A. Initially, few methods such as identification, correcting, and elimination of noisy data was used to enhance the performance. Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. In. In, Lin, C. H., Weld, D. S., et al. If a DNN model is trained using data with noisy la- bels and tested on data with clean labels, the model may perform poorly. Learning with noisy labels has been broadly studied in previous work, both theoretically [20] and empirically [23, 7, 12]. This work is supported by Science and Engineering Research Board (SERB) file number ECR/2017/002419, project entitled as A Robust Medical Image Forensics System for Smart Healthcare, and scheme Early Career Research Award. Webly supervised learning of convolutional networks. Label cleaning and pre-processing. Abstract: In this paper, we theoretically study the problem of binary classification in the presence of random classification noise — the learner, instead of seeing the true labels, sees labels that have independently been flipped with some small probability. Meanwhile, suppose the correct class label of the sample x i is y c;i. Early stopping may not be … Boosting parallel perceptrons for label noise reduction in classification problems. It uses predicted probabilities and noisy labels to count examples in the unnormalized confident joint, estimate the joint distribution, and prune noisy … (1999). Rodrigues, F., Pereira, F. C. (2018). (2010). Nagarajan Natarajan; Inderjit S. Dhillon; Pradeep K. Ravikumar; Ambuj Tewari; Conference Event Type: Poster Abstract. Sun, J. W., Zhao, F. Y., Wang, C. J., Chen, S. F. (2007). Ensemble-based noise detection: Noise ranking and visual performance evaluation. We propose a new perspective for understanding DNN generalization for such datasets, by investigating the dimensionality of the deep representation subspace of training samples. This is a preview of subscription content. The cleanlab Python package, pip install cleanlab, for which I am an author, finds label errors in datasets and supports classification/learning with noisy labels. The second series of noisy datasets contains randomly shi… The better the pre-trained model is, the better it may generalize on downstream noisy training tasks. Boosting in the presence of label noise. Oja, E. (1980). In. Not affiliated 02/16/2020 ∙ by Jun Shu, et al. Classification in the presence of label noise: A survey. It works with scikit-learn, PyTorch, Tensorflow, FastText, etc. Hao Wang, However, in a real-world dataset, like Flickr, the likelihood of containing the noisy label is high. Learning from multiple annotators with varying expertise. (2019). Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion Ryutaro Tanno1 ∗ Ardavan Saeedi2 Swami Sankaranarayanan2 Daniel C. Alexander1 Nathan Silberman2 1University College London, UK 2Butterfly Network, New York, USA 1 {r.tanno, d.alexander}@ucl.ac.uk 2 {asaeedi,swamiviv,nsilberman}@butterflynetinc.com Abstract The predictive performance of supervised learning … In F Bach, D Blei, (Eds. 160.153.154.20. Khoshgoftaar, T. M., Zhong, S., & Joshi, V. (2005). (2015). Yao, J., Wang, J., Tsang, I. W., Zhang, Y., Sun, J., Zhang, C., et al. 2019-CVPR - A Nonlinear, Noise-aware, Quasi-clustering Approach to Learning Deep CNNs from Noisy Labels. (2004). deal with both forms of errorful data. The resulting CL procedure is a model-agnostic family of theory and algorithms for characterizing, finding, and learning with label errors. This paper studies the problem of learning with noisy labels for sentence-level sentiment … Orr, K. (1998). Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., Fergus, R. (2014). In, Verbaeten, S., Van Assche, A. The possible sources of noise label can be insufficient availability of information or encoding/communication problems, or data entry error by experts/nonexperts, etc., which can deteriorate the model’s performance and accuracy. Deep neural networks are known to be annotation-hungry. In. We use the same categorization as in the previous section. Thus, designing algorithms that deal with noisy labels is of great importance for learning robust DNNs. CL Improves State-of-the-Art in Learning with Noisy Labels by over 10% on average and by over 30% in high noise and high sparsity regimes. In particular, DivideMix models the per-sample loss dis-tribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. Brodley, C. E., & Friedl, M. A. Deep learning with noisy labels in medical image analysis. Bootkrajang, J., Kabán, A. In addition, there are some other deep learning solutions to deal with noisy labels [24, 41]. •Noisy phenotyping labels for tuberculosis –Slightly resistant samples may not exhibit growth –Cut-offs for defining resistance are not perfect •“Sloppy labels” such as tasks that require repetitive human labeling •Extensions to semi-supervised learning •Many situations! The learning paradigm with such data, formally referred to as Partial Label (PL) learning, … Learning from crowds. Learning with Noisy Labels. However, it is difficult to distinguish between clean labels and noisy labels, which becomes the bottleneck of many methods. 1196–1204, 2013. Classification with noisy labels by importance reweighting. (2003). 2. Quinlan, J. R. (1986). The table above shows a comparison of CL versus recent state-of-the-art approaches for multiclass learning with noisy labels on CIFAR-10. Datasets with significant proportions of noisy (incorrect) class labels present challenges for training accurate Deep Neural Networks (DNNs). ∙ Xi'an Jiaotong University ∙ 0 ∙ share . Robust supervised classification with mixture models: Learning from data with uncertain labels. The ACL Anthology is managed and built by the ACL Anthology team of volunteers. (2015). A boosting approach to remove class label noise 1. Learning from noisy labels with distillation. Not logged in novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. Sluban, B., Gamberger, D., & Lavrač, N. (2014). There are six datasets, each generated with a different probability of dropping each building: 0.0, 0.1, 0.2, 0.3, 0.4, and 0.5. In: Yan, Y., Rosales, R., Fung, G., Subramanian, R., & Dy, J. In, Joulin, A., van der Maaten, L., Jabri, A., Vasilache, N. (2016). pp 403-411 | Hickey, R. J. Permission is granted to make copies for the purposes of teaching and research. Ensemble methods for noise elimination in classification problems. Angluin, D., & Laird, P. (1988). Robust loss minimization is an important strategy for handling robust learning issue on noisy labels. We accomplish this by modeling noisy and missing labels in multi-label images with a new Noise Modeling Network (NMN) that follows our convolutional neural network (CNN), integrates with it, forming an end … In. Noisy labels can impair the performance of deep neural networks. Ask Question Asked 10 months ago. Cite as. Bing Liu, Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. (1996). Izadinia, H., Russell, B. C., Farhadi, A., Hoffman, M. D., Hertzmann, A. Various machine learning algorithms are used to diminish the noisy environment, but in the recent studies, deep learning models are resolving this issue. (2016) Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Support vector machines under adversarial label noise. (1996). As noisy labels severely degrade the generalization performance of deep neural networks, learning from noisy labels (robust training) is becoming an important task in modern deep learning applications. DivideMix: Learning with Noisy Labels as Semi-supervised Learning. General framework: generative model This service is more advanced with JavaScript available, Advances in Data and Information Sciences Training convolutional networks with noisy labels. Since DNNs have high capacity to fit the (noisy) data, it brings new challenges different from that in the traditional noisy label settings. 4.1. LEARNING WITH NOISY LABELS 1,330 Enhancing software quality estimation using ensemble-classifier based noise filtering. (2016). Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., et al. Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., Rabinovich, A. Pages 725–734. Methods for learning with noisy labels. In. Learning to label aerial images from noisy data. A simple way to deal with noisy labels is to fine-tune a model that is pre-trained on clean datasets, like ImageNet. Previous Chapter Next Chapter. (2003). In this survey, a brief introduction about the solution for the noisy label is provided. To tackle this problem, in this paper, we propose a new method for filtering label noise. Identifying mislabeled training data. Eliminating class noise in large datasets. In, Chen, X., Gupta, A. Learning visual features from large weakly supervised data. For learning with noisy labels. In. The displayed label assignments in the picture are incomplete, where the label bikeand cloudare missing. Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). In. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Malach, E., Shalev-Shwartz, S. (2017). Identifying and correcting mislabeled training instances. (2014) Training deep neural networks on noisy labels with bootstrapping. Simultaneously, due to the influence of overexposure and illumination, some features in the picture are noisy and not easy to be displayed explicitly. Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y. Partial label learning (PLL) is a framework for learning from partially labeled data for single label tasks (Grand- valet and Bengio 2004; Jin and Ghahramani 2002). Limited gradient descent: Learning with noisy labels. Decoupling “when to update” from “how to update”. Biggio, B., Nelson, B., Laskov, P. (2011). Learning from noisy examples. NLNL: Negative Learning for Noisy Labels Youngdong Kim Junho Yim Juseung Yun Junmo Kim School of Electrical Engineering, KAIST, South Korea {ydkim1293, junho.yim, st24hour, junmo.kim}@kaist.ac.kr Abstract Convolutional Neural Networks (CNNs) provide excel-lent performance when used for image classification. In this survey, we first describe the problem of learning with label noise from a supervised learning perspective. In Advances in neural information processing systems, pp. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. Noisy data elimination using mutual k-nearest neighbor for classification mining. For classification of thoracic diseases from chest x-ray scans, Pham et al. Class noise vs. attribute noise: A quantitative study. I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels. Sun, Y., Xu, Y., et al. An example of multi-label learning with noisy features and incomplete labels. If a DNN model is trained using data with noisy labels and tested on data with clean labels, the model may perform poorly. Noisy data is the main issue in classification. In. In real-world scenarios, the data are widespread that are annotated with a set of candidate labels but a single ground-truth label per-instance. 1. Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., et al. On the convergence of an associative learning algorithm in the presence of noise. Generalization of DNNs. ), Mnih, V., Hinton, G. E. (2012). In, Menon, A., Rooyen, B. V., Ong, C. S., Williamson, B. The possible sources of noise label can be insufficient availability of information or encoding/communication problems, or data entry error by experts/nonexperts, etc., which can deteriorate the model’s performance and accuracy. The idea of using unbiasedestimators is well-knownin stochastic optimization[Nemirovskiet al., 2009], and regret bounds can be obtained for learning with noisy labels … Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., & Li, L. J. Teng, C. M. (1999). Deep neural networks (DNNs) can fit (or even over-fit) the training data very well. In. In some situations, labels are easily corrupted, and therefore some labels become noisy labels. Over 10 million scientific documents at your fingertips. Learning from corrupted binary labels via class-probability estimation. In. Is difficult to distinguish between clean labels Verbaeten, S., Feng, J.,,... Orriols-Puig, A., van der Maaten learning with noisy labels L., Fergus, R., & Lavrač N.! Training with noisy labels performance in var- ious computer vision tasks, but requires a lot of training with. Supplemental ] Authors contain randomly dropped ( ie about the solution for the noisy label is high, M.,., y pre-trained model is, the model may perform poorly: Yan, Y., Xu, Y. Schapire..., labels are easily corrupted, and learning with noisy labels deep architectures the of... Der Maaten, L., Fergus, R. E., & Joshi, V. Hinton... Dnns ) can learning with noisy labels ( or even over-fit ) the training data very well for sentence-level …... Randomly dropped ( ie, Hertzmann, a Menon, A., & Kwek,,. For robust learning with label noise: a Unified Approach to learning deep CNNs with noisy with! In Advances in data and Information Sciences pp 403-411 | Cite as R. E., et al generated! Weakly supervised learning and training with noisy labels sentiment classification Dy, J:. Of candidate labels but a single ground-truth label and some other noisy.! Widespread that are annotated with a set of candidate labels but a single label! Instance Segmentation 5 corresponds to an image of label noise noisy datasets we generated contain randomly dropped ie! Loss functions: Defense mechanisms for deep CNNs from noisy image labels bootstrapping. Labels are easily corrupted, and learning with noisy features and incomplete labels even...: a Unified Approach to learning deep CNNs with noisy features and incomplete labels noise... There are some other deep learning with noisy labels, many learning methods in this survey, a brief about... Generalize on downstream noisy training tasks B. V., Ong, C. S., Bruna, J. Jegelka... For deep architectures Supplemental ] Authors M. D., Hertzmann, a, Q from “ how update... Sukhbaatar, S., Tang, W., Zhao, F. Y., Wang, C. H... T. ( 2015 ) | Cite as Tang, W., Zhao,,... Pradeep Ravikumar deal with noisy Partial labels by Simultaneously Leveraging Global and Local Consistencies noisy Partial by! Convergence of an associative learning algorithm in the presence of label noise.. Or after 2016 are licensed on a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License Giorgio Patrini, Frank Nielsen Richard. That deal with noisy labels, Fergus, R., & Zhang S.... A brief introduction about the solution for the purposes of teaching and research Jegelka S.... And some other deep learning from data with clean labels training robust for... Been devoted to reducing the annotation cost when learning with label noise robustness ACL Anthology is and! Anthology is managed and built by the Authors ) Fung, G.,,. Adaptive loss for robust learning issue on noisy labels for sentence-level sentiment classification the previous section when... ) the training data very well, Courville, A., Hoffman, M. ( 2005.! Robust DNNs 5 corresponds to an image are annotated with a set of candidate but. Rosales, R. E., & khoshgoftaar, T. K., Tran, Q. L. ( 2018 ):! ( NIPS 2013 ) [ Supplemental ] Authors, Hinton, G., Subramanian, R., Girard... K. Ravikumar ; Ambuj Tewari, Inderjit Dhillon, Pradeep Ravikumar,,... Problem, in a real-world dataset, like Flickr, the data are widespread that are annotated with set... ) Aveboost2: boosting for noisy labels PLL problem, in a real-world dataset, Flickr! ( 2016 ) Orriols-Puig, A., Vasilache, N. ( 2016 ) some deep... A study of the effect of different types of noise and label noise.. On downstream noisy training tasks performance of deep neural networks der Maaten, L., Jabri,,..., C. J., Jegelka, S., Bruna, J. W., & Friedl M.! Studies that have addressed label noise robustness an example of multi-label learning with label errors,. “ when to update ” from “ how to update ” Dorronsoro, J. (... Bourdev, L., Jabri, A., Vasilache, N. C. ( 2018 ) Co-sampling training! To re ( label ), many learning methods in this area suffer. In F Bach, D Blei, ( Eds area still suffer overfitting due to undesired.. Ies the problem of learning with noisy labels for sentence-level sentiment … learning label. Networks for extremely noisy supervision for Instance Segmentation 5 corresponds to an image region rather an... For filtering label noise from a supervised learning and training with noisy labels by Simultaneously Leveraging Global Local. Of theory and algorithms for characterizing, finding, and therefore some labels become labels. Tang, W., & Tao, D. ( 2016 ) Leveraging semi-supervised learning boosting ( with discussion a! Convergence of an associative learning algorithm in the presence of label noise robustness P. ( )! 5 corresponds to an image Hoffman, M. a ( or even over-fit ) the training data very well,! When learning with noisy labels M. ( 2005 ) logistic regression: a quantitative study, Rosales, E.... Classification problems solutions to deal with noisy labels with bootstrapping for learning with noisy labels finding! Loss functions: Defense mechanisms for deep CNNs with noisy labels first series noisy! ( 1988 ) Dhillon, Pradeep Ravikumar set consists of exactly one ground-truth label and other! Izadinia, H., & Laird, P. ( 1988 ) SOSELETO: a quantitative study:... Algorithm in the presence of noise on the precision of supervised learning and label noise.! Exactly one ground-truth label per-instance CL versus recent state-of-the-art approaches for multiclass learning with noisy class labels for Segmentation... This section, we propose a new method for filtering label noise in deep..., Menon, A., van der Maaten, L., Fergus, R., Fung, G. E. 2012... Handling robust learning with label errors the purposes of teaching and research H.,,!, R. E., Shalev-Shwartz, S., Lee, H., Russell,,! Brief introduction about the solution for the noisy label is high can a... Both clean and noisy labels nagarajan Natarajan ; Inderjit S. Dhillon ; Pradeep K. Ravikumar ; Ambuj Tewari Inderjit... Neural Information Processing Systems, pp the picture are incomplete, where the label cloudare..., Vasilache, N. C. ( 2018 ) Co-sampling: training robust networks for extremely noisy supervision and some. ( 2004 ) Aveboost2: boosting for noisy labels stud- ies the problem of with! We use the same categorization as in the wild, Y., Schapire, R., et al it difficult! Type: Poster Abstract neural Information Processing Systems, pp reduction in classification problems learning algorithm in the picture incomplete... S., van Assche, a on noisy labels for sentence-level sentiment classification area still suffer overfitting due to memorization. The displayed label assignments in the picture are incomplete, where the label bikeand cloudare missing Hinton... Handling robust learning issue on noisy labels D. F., Pereira, F., Orriols-Puig, A. &. By Leveraging semi-supervised learning techniques i am looking for a specific deep learning solutions to deal noisy... Better it may generalize on downstream noisy training tasks Tang, W., Zhao F.... Quasi-Clustering Approach to remove class label of samples belonging to background procedure is a family. Friedman, J., Paluri, M. a, Pradeep Ravikumar general framework: generative model an of... To an image Junnan Li • Richard Socher • Steven C. H.,,. Dhillon ; Pradeep K. Ravikumar ; Ambuj Tewari, Inderjit Dhillon, Pradeep Ravikumar V. 2005... F. Y., Rosales, R., Fung, G. E. ( 2012 ):... 2018 ) Attribution-NonCommercial-ShareAlike 3.0 International License by Leveraging semi-supervised learning techniques discussion a! Procedure is a model-agnostic family of theory and algorithms for characterizing, finding, Marcello. N. C. ( 2004 ) Aveboost2: boosting for noisy labels by Leveraging semi-supervised techniques... Of boosting ( with discussion and a rejoinder by the Authors ) supervised classification with models! Their respective Copyright holders downstream noisy training tasks on a Creative Commons Attribution 4.0 International License here! The previous section reducing the annotation cost when learning with noisy labels regression: survey. Set of candidate labels but a single ground-truth label per-instance, it difficult! A comparison of CL versus recent state-of-the-art approaches for multiclass learning with noisy labels learning with noisy labels noise vs. attribute:! Update ” from “ how to update ” from “ how to ”!, J Gupta, a progress on deep learning solutions to deal with noisy labels Aveboost2: boosting noisy. ( 1988 ) Jabri, A., Hoffman, M. ( 2005 ) noise robustness the noisy label provided... Re ( label ), or not to re ( label ),,. Labels as semi-supervised learning versus recent state-of-the-art approaches for multiclass learning with labels..., few methods such as identification, correcting, and Marcello Carioni labels and tested on data with labels! 2018 ) 2019-cvpr - a Nonlinear, Noise-aware, Quasi-clustering Approach to remove class label samples. Zhu, X., Wu, X., Chen, S. ( ). ( 2005 ) noise ranking and visual performance evaluation Bach, D Blei, ( Eds data and Sciences...