skip to main content
Language:
Search Limited to: Search Limited to: Resource type Show Results with: Show Results with: Search type Index

Wide and deep neural networks achieve consistency for classification

Proceedings of the National Academy of Sciences - PNAS, 2023-04, Vol.120 (14), p.e2208779120-e2208779120 [Peer Reviewed Journal]

Copyright National Academy of Sciences Apr 4, 2023 ;Copyright © 2023 the Author(s). Published by PNAS. 2023 ;ISSN: 0027-8424 ;EISSN: 1091-6490 ;DOI: 10.1073/pnas.2208779120 ;PMID: 36996114

Full text available

Citations Cited by
  • Title:
    Wide and deep neural networks achieve consistency for classification
  • Author: Radhakrishnan, Adityanarayanan ; Belkin, Mikhail ; Uhler, Caroline
  • Subjects: Artificial neural networks ; Classification ; Classifiers ; Consistency ; Kernels ; Machine Learning ; Neural networks ; Neural Networks, Computer ; Physical Sciences ; Statistical analysis ; Taxonomy ; Training
  • Is Part Of: Proceedings of the National Academy of Sciences - PNAS, 2023-04, Vol.120 (14), p.e2208779120-e2208779120
  • Description: While neural networks are used for classification tasks across domains, a long-standing open problem in machine learning is determining whether neural networks trained using standard procedures are consistent for classification, i.e., whether such models minimize the probability of misclassification for arbitrary data distributions. In this work, we identify and construct an explicit set of neural network classifiers that are consistent. Since effective neural networks in practice are typically both wide and deep, we analyze infinitely wide networks that are also infinitely deep. In particular, using the recent connection between infinitely wide neural networks and neural tangent kernels, we provide explicit activation functions that can be used to construct networks that achieve consistency. Interestingly, these activation functions are simple and easy to implement, yet differ from commonly used activations such as ReLU or sigmoid. More generally, we create a taxonomy of infinitely wide and deep networks and show that these models implement one of three well-known classifiers depending on the activation function used: 1) 1-nearest neighbor (model predictions are given by the label of the nearest training example); 2) majority vote (model predictions are given by the label of the class with the greatest representation in the training set); or 3) singular kernel classifiers (a set of classifiers containing those that achieve consistency). Our results highlight the benefit of using deep networks for classification tasks, in contrast to regression tasks, where excessive depth is harmful.
  • Publisher: United States: National Academy of Sciences
  • Language: English
  • Identifier: ISSN: 0027-8424
    EISSN: 1091-6490
    DOI: 10.1073/pnas.2208779120
    PMID: 36996114
  • Source: Geneva Foundation Free Medical Journals at publisher websites
    MEDLINE
    PubMed Central
Back to results list
INSPIRE LIBRARY - TON DUC THANG UNIVERSITY (84-028) 37 755 057 Feedback
19 Nguyen Huu Tho St. Dist.7, HCM thuvien@tdtu.edu.vn

Copyright © 2017 INSPIRE LIBRARY - TON DUC THANG UNIVERSITY

Searching Remote Databases, Please Wait