The Story Picturing Engine: Finding Elite Images to Illustrate a Story
Using Mutual Reinforcement

Dhiraj Joshi, James Z. Wang, Jia Li
The Pennsylvania State University, University Park, PA 16802

In this paper, we present an approach towards automated story picturing based on mutual reinforcement principle. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We also present a human behavior model based on a discrete state Markov process which captures the intuition for our technique. Experimental results have demonstrated the e ectiveness of our scheme. Categories and Subject Descriptors: H.3.3 [Information Search and Retrieval]: retrieval models; search process; selection process.

Full Paper in Color
(PDF, 3MB)

Try the Story Picturing Engine Demo

Other On-line Info

Citation: Dhiraj Joshi, James Z. Wang and Jia Li, ``The Story Picturing Engine: Finding Elite Images to Illustrate a Story Using Mutual Reinforcement,'' Proc. 6th International Workshop on Multimedia Information Retrieval, in conjunction with ACM Multimedia, pp. 119-126, New York, NY, ACM, October 2004.

Copyright 2004 ACM. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the ACM.

Last Modified: August 13, 2004