ICBO_2018_70: Improving Convergence Rates of Deep Learning for Very Small Image Training Sets

Typical visualization tasks in the domain of big image datasets include: image retrieval, clustering, and segmentation. Recently, there has been a tremendous progress in solving these tasks using deep convolutional neural networks (CNNs). CNNs typically require large training sets of manually annotated images for deep learning, which, however, is often impossible to provide in various applications, including image-based biological and medical research. This talk will address one way to address this critical issue – a new CNN learning approach, based on second-order methods, aimed at improving: a) Convergence rates of existing gradient-based methods, and b) Robustness to the choice of learning hyper-parameters (e.g., learn- ing rate). Our approach simultaneously computes both gradients and second derivatives of the CNN's learning objective, and performs second-order back-propagation. In comparison with standard gradient-based deep learning, our evaluation demonstrates that we achieve faster convergences rates, and converge to better optima leading to better performance under a budgeted time for learning.