Tagging over Time: Real-world Image Annotation by Lightweight Meta-learning

Tagging over Time: Real-world Image Annotation by Lightweight Meta-learning

Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang
The Pennsylvania State University, University Park, PA 16802
Abstract:

Automatic image annotation has been a hot-pursuit among multimedia researchers of late. Modest performance guarantees and limited adaptability often restrict its applicability to real-world settings. We propose tagging over time (T/T) to push the technology toward real-world applicability. Of particular interest are online systems that receive user-provided images and feedback over time, with user focus possibly changing and evolving. The T/T framework consists of a principled probabilistic approach to metalearning, which acts as a go-between for a black-box annotation system and the users. Inspired by inductive transfer, the approach attempts to harness available information, including the black-box models performance, the image representations, and the WordNet ontology. Being computationally lightweight, this meta-learner efficiently re-trains over time, to improve and/or adapt to changes. The black-box annotation model is not required to be re-trained, allowing computationally intensive algorithms to be used. We experiment with standard image datasets and real-world data streams, using two existing annotation systems as blackboxes. Both batch and online annotation settings are experimented with. It is observed that the addition of this metalearning layer produces much improved results that outperform best-known results. For the online setting, the T/T approach produces progressively better annotation with time, significantly outperforming the black-box as well as the static form of the meta-learner, on real-world data.

Full Paper in Color

(PDF, 0.85MB)

On-line Info

Citation: Ritendra Datta, Dhiraj Joshi, Jia Li and James Z. Wang, ``Tagging over Time: Real-world Image Annotation by Lightweight Meta-learning,'' Proceedings of the ACM Multimedia Conference, pp. 393-402, ACM, Augsburg, Germany, September 2007.

Copyright 2007 ACM. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the ACM.

Last Modified: October 1, 2007
© 2007