Targeted Data-driven Regularization for Out-of-Distribution Generalization
Mohammad Mahdi Kamani, Sadegh Farhang, Mehrdad Mahdavi and James Z. Wang
The Pennsylvania State University, University Park, Pennsylvania, USA
Abstract:
Due to biases introduced by large real-world datasets, deviations of
deep learning models from their expected behavior on
out-of-distribution test data are worrisome. Especially when data come
from imbalanced or heavy-tailed label distributions, or minority
groups of a sensitive feature. Classical approaches to address these
biases are mostly data- or application-dependent, hence are burdensome
to tune. Some meta-learning approaches, on the other hand, aim to
learn hyperparameters in the learning process using different
objective functions on training and validation data. However, these
methods suffer from high computational complexity and are not scalable
to large datasets. In this paper, we propose a unified data-driven
regularization approach to learn a generalizable model from biased
data. The proposed framework, named as targeted data-driven
regularization (TDR), is model- and dataset-agnostic, and employs a
target dataset that resembles the desired nature of test data in order
to guide the learning process in a coupled manner. We cast the problem
as a bilevel optimization and propose an efficient stochastic gradient
descent based method to solve it. The framework can be utilized to
alleviate various types of biases in real-world applications. We
empirically show, on both synthetic and real-world datasets, the
superior performance of TDR for resolving issues stem from these
biases.
Full Paper
(high-quality PDF, 3MB)
More information
Citation:
Mohammad M. Kamani, Sadegh Farhang, Mehrdad Mahdavi and James Z. Wang,
``Targeted Data-driven Regularization for Out-of-Distribution
Generalization,'' Proceedings of the ACM SIGKDD Conference on
Knowledge Discovery and Data Mining (KDD), pp. 882-891, virtual,
August 2020.
© 2020 ACM. Personal use of this material is permitted. However,
permission to reprint/republish this material for advertising or
promotional purposes or for creating new collective works for resale
or redistribution to servers or lists, or to reuse any copyrighted
component of this work in other works must be obtained from the ACM.
Last Modified:
June 18, 2020
© 2020