A Simulated Annealing Based Inexact Oracle for Wasserstein Loss Minimization

Jianbo Ye, James Z. Wang and Jia Li

The Pennsylvania State University, USA
Abstract:

Learning under a Wasserstein loss is an emerging research topic. We call collectively the problems formulated under this framework Wasserstein loss minimization (WLM). One important appeal of WLM is the innate capability to account for the similarity between atoms or features, while the usual decomposable losses treat the features as separate dimensions. Despite being conceptually simple, WLM problems are computationally challenging because they involve minimizing over functions of quantities (i.e. Wasserstein distances) that themselves require numerical algorithms to compute. Most recent approaches use an entropy-type barrier function on the primal domain, resulting in fast optimization with a smoothed Wasserstein loss. In this paper, we introduce a new technique based on simulated annealing, a stochastic approach that implicitly uses a distance-matrix-dependent barrier on the dual domain. With this approach, we can draw on a rich body of work on Markov chain Monte Carlo. We have developed a Gibbs sampler to approximate effectively and efficiently the partial gradients of a sequence of Wasserstein losses. Our new approach has the advantages of numerical stability and readiness for warm starts. These characteristics are valuable for WLM problems that often require multiple levels of iterations in which the oracle for computing the value and gradient of a loss function is embedded. We applied the method to optimal transport with Coulomb cost and the Wasserstein non-negative matrix factorization problem, and made comparisons with the existing method of entropy regularization.


Full Paper
(PDF, 1.6MB)



Citation: Jianbo Ye, James Z. Wang and Jia Li, ``A Simulated Annealing Based Inexact Oracle for Wasserstein Loss Minimization,'' Proceedings of the International Conference on Machine Learning, 12 pages, 2017.

© 2017. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the authors.

Last Modified: May 13, 2017
© 2017