Unlocking the Emotional World of Visual Media:
An Overview of the Science, Research, and Impact
of Understanding Emotion
James Z. Wang (1), Sicheng Zhao (2), Chenyan Wu (1),
Reginald B. Adams, Jr. (1), Michelle G. Newman (1), Tal Shafir (3), Rachelle Tsachor (4)
(1) The Pennsylvania State University, USA
(2) Tsinghua University, China
(3) University of Haifa, Israel
(4) University of Illinois, Chicago, USA
Drawing insights from psychology, engineering, and the arts, this
article provides a comprehensive overview of the field of emotion
analysis in visual media, and discusses the latest research, systems,
challenges, ethical implications, and potential impact of artificial
emotional intelligence on society.
Abstract:
The emergence of artificial emotional intelligence technology is
revolutionizing the fields of computers and robotics, allowing for a
new level of communication and understanding of human behavior that
was once thought impossible. Whereas recent advancements in deep
learning have transformed the field of computer vision, automated
understanding of evoked or expressed emotions in visual media remains
in its infancy. This foundering is due in part to the lack of
consensus among scientists regarding a definition of ``emotion'' as
well as the subjectivity of emotions and the subtle distinctions
between them. In this paper, we provide a comprehensive,
multidisciplinary overview of the field of emotion analysis in visual
media, drawing on insights from psychology, engineering, and the
arts. We begin by exploring the psychological foundations of emotion
and the computational principles that underpin the understanding of
emotions from images and videos. We then review the latest research
and systems that have been developed in the field, highlighting the
most promising approaches. We also discuss the current technological
challenges and limitations of emotion analysis, and emphasize the need
for further research and development in this area. We argue that this
is a ``Holy Grail'' research problem in computing and outline some of
the key directions for future research. Finally, we examine the
ethical implications of emotion-understanding technologies and
consider the potential impact of these technologies on
society. Overall, this article aims to provide readers with a deeper
understanding of the field of emotion analysis in visual media, and to
inspire further research and development in this exciting and rapidly
evolving field.
Full Paper
(high-resolution PDF, 19MB; email jwang (at) ist.psu.edu to request)
Full Paper
(reduced-resolution PDF, 3MB)
More information
Citation:
James Z. Wang, Sicheng Zhao, Chenyan Wu, Reginald B. Adams, Jr.,
Michelle G. Newman, Tal Shafir and Rachelle Tsachor, ``Unlocking the
Emotional World of Visual Media: An Overview of the Science, Research,
and Impact of Understanding Emotion,'' Proceedings of the IEEE, 2023,
under second-round review.
© 2023 IEEE. Personal use of this material is permitted. However,
permission to reprint/republish this material for advertising or
promotional purposes or for creating new collective works for resale
or redistribution to servers or lists, or to reuse any copyrighted
component of this work in other works must be obtained from the IEEE.
Last Modified:
March 8, 2023
© 2023