The Story Picturing Engine - A System for Automatic Text Illustration

Dhiraj Joshi, James Z. Wang, Jia Li
The Pennsylvania State University, University Park, PA 16802
Abstract:

In this paper, we present an unsupervised approach towards automated story picturing. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We have implemented the methods in our Story Picturing Engine (SPE) system. Experiments on two large-scale image databases are reported. A user study has been performed and statistical analysis of the results has been presented.


Full Paper in Color

(PDF, 0.8MB)

Try the Story Picturing Engine Demo

On-line Info


Citation: Dhiraj Joshi, James Z. Wang and Jia Li, ``The Story Picturing Engine - A System for Automatic Text Illustration,'' ACM Transactions on Multimedia Computing, Communications and Applications, vol. 2, no. 1, pp. 68-89, 2006.

Copyright 2005 ACM. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the ACM.

Last Modified: Nov 1, 2005
© 2005