Language:

Minimizing finite sums with the stochastic average gradient

Mathematical programming, 2017-03, Vol.162 (1-2), p.83-112 [Peer Reviewed Journal]

Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2016 ;Mathematical Programming is a copyright of Springer, 2017. ;Distributed under a Creative Commons Attribution 4.0 International License ;ISSN: 0025-5610 ;EISSN: 1436-4646 ;DOI: 10.1007/s10107-016-1030-6 ;CODEN: MHPGA4

Full text available

Citations Cited by

Actions
1. Add to My Research
2. Remove from My Research
3. E-mail
4. Print
5. Permalink
6. Citation
7. EasyBib
8. EndNote
9. RefWorks
10. Delicious
11. Export RIS
12. Export BibTeX

Title:
Minimizing finite sums with the stochastic average gradient
Author: Schmidt, Mark ; Le Roux, Nicolas ; Bach, Francis
Subjects: Algorithms ; Calculus of Variations and Optimal Control; Optimization ; Combinatorics ; Computation ; Computer Science ; Convergence ; Convex analysis ; Full Length Paper ; Machine Learning ; Mathematical analysis ; Mathematical and Computational Physics ; Mathematical Methods in Physics ; Mathematical models ; Mathematics ; Mathematics and Statistics ; Mathematics of Computing ; Methods ; Numerical Analysis ; Optimization ; Optimization and Control ; Regression analysis ; Sag ; Sampling ; Statistics ; Stochasticity ; Strategy ; Studies ; Texts ; Theoretical
Is Part Of: Mathematical programming, 2017-03, Vol.162 (1-2), p.83-112
Description: We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method’s iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from O ( 1 / k ) to O (1 / k ) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O (1 / k ) to a linear convergence rate of the form O ( ρ k ) for ρ < 1 . Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. This extends our earlier work Le Roux et al. (Adv Neural Inf Process Syst, 2012 ), which only lead to a faster rate for well-conditioned strongly-convex problems. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.
Publisher: Berlin/Heidelberg: Springer Berlin Heidelberg
Language: English
Identifier: ISSN: 0025-5610
EISSN: 1436-4646
DOI: 10.1007/s10107-016-1030-6
CODEN: MHPGA4
Source: Hyper Article en Ligne (HAL) (Open Access)
ProQuest Central

Back to results list


INSPIRE LIBRARY - TON DUC THANG UNIVERSITY	(84-028) 37 755 057	Feedback
19 Nguyen Huu Tho St. Dist.7, HCM	thuvien@tdtu.edu.vn	Feedback

Minimizing finite sums with the stochastic average gradient

Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2016 ;Mathematical Programming is a copyright of Springer, 2017. ;Distributed under a Creative Commons Attribution 4.0 International License ;ISSN: 0025-5610 ;EISSN: 1436-4646 ;DOI: 10.1007/s10107-016-1030-6 ;CODEN: MHPGA4

Searching Remote Databases, Please Wait