skip to main content
Language:
Search Limited to: Search Limited to: Resource type Show Results with: Show Results with: Search type Index

Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data

IEEE transaction on neural networks and learning systems, 2020-09, Vol.31 (9), p.3400-3413

ISSN: 2162-237X ;EISSN: 2162-2388 ;DOI: 10.1109/TNNLS.2019.2944481 ;PMID: 31689214 ;CODEN: ITNNAL

Digital Resources/Online E-Resources

Citations Cited by
  • Title:
    Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data
  • Author: Sattler, Felix ; Wiedemann, Simon ; Muller, Klaus-Robert ; Samek, Wojciech
  • Subjects: Data models ; Deep learning ; Distributed databases ; distributed learning ; efficient communication ; federated learning ; privacy-preserving machine learning ; Protocols ; Servers ; Training ; Training data
  • Is Part Of: IEEE transaction on neural networks and learning systems, 2020-09, Vol.31 (9), p.3400-3413
  • Description: Federated learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning, however, comes at the cost of a significant communication overhead during training. To address this problem, several compression methods have been proposed in the distributed training literature that can reduce the amount of required communication by up to three orders of magnitude. These existing methods, however, are only of limited utility in the federated learning setting, as they either only compress the upstream communication from the clients to the server (leaving the downstream communication uncompressed) or only perform well under idealized conditions, such as i.i.d. distribution of the client data, which typically cannot be found in federated learning. In this article, we propose sparse ternary compression (STC), a new compression framework that is specifically designed to meet the requirements of the federated learning environment. STC extends the existing compression technique of top-k gradient sparsification with a novel mechanism to enable downstream compression as well as ternarization and optimal Golomb encoding of the weight updates. Our experiments on four different learning tasks demonstrate that STC distinctively outperforms federated averaging in common federated learning scenarios. These results advocate for a paradigm shift in federated optimization toward high-frequency low-bitwidth communication, in particular in the bandwidth-constrained learning environments.
  • Publisher: IEEE
  • Language: English
  • Identifier: ISSN: 2162-237X
    EISSN: 2162-2388
    DOI: 10.1109/TNNLS.2019.2944481
    PMID: 31689214
    CODEN: ITNNAL
  • Source: IEEE Open Access Journals
Back to results list
INSPIRE LIBRARY - TON DUC THANG UNIVERSITY (84-028) 37 755 057 Feedback
19 Nguyen Huu Tho St. Dist.7, HCM thuvien@tdtu.edu.vn

Copyright © 2017 INSPIRE LIBRARY - TON DUC THANG UNIVERSITY

Searching Remote Databases, Please Wait