Toward Rapid Stroke Diagnosis with Multimodal Deep Learning

Mingli Yu (1), Tongan Cai (1), Xiaolei Huang (1), Kelvin Wong (2), John Volpi (3), James Z. Wang (1), and Stephen T.C. Wong (2)

(1) The Pennsylvania State University, University Park, Pennsylvania, USA
(2) TT and WF Chao Center for BRAIN & Houston Methodist Cancer Center, Houston Methodist Hospital, Houston, Texas, USA
(3) Eddy Scurlock Comprehensive Stroke Center, Department of Neurology, Houston Methodist Hospital, Houston, Texas, USA Abstract:

Stroke is a challenging disease to diagnose in an emergency room (ER) setting. While an MRI scan is very useful in detecting ischemic stroke, it is usually not available due to space constraint and high cost in the ER. Clinical tests like the Cincinnati Pre-hospital Stroke Scale (CPSS) and the Face Arm Speech Test (FAST) are helpful tools used by neurologists, but there may not be neurologists immediately available to conduct the tests. We emulate CPSS and FAST and propose a novel multimodal deep learning framework to achieve computer-aided stroke presence assessment over facial motion weaknesses and speech inability for patients with suspicion of stroke showing facial paralysis and speech disorders in an acute setting. Experiments on our video dataset collected on actual ER patients performing specific speech tests show that the proposed approach achieves diagnostic performance comparable to that of ER doctors, attaining a 93.12% sensitivity rate while maintaining 79.27% accuracy. Meanwhile, each assessment can be completed in less than four minutes. This demonstrates the high clinical value of the framework. In addition, the work, when deployed on a smartphone, will enable selfassessment by at-risk patients at the time when stroke-like symptoms emerge.

Full Paper
(PDF, 0.4MB)

More information


Citation: Mingli Yu, Tongan Cai, Xiaolei Huang, Kelvin Wong, John Volpi, James Z. Wang and Stephen T.C. Wong, ``Toward Rapid Stroke Diagnosis with Multimodal Deep Learning,'' Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions, pp. 616-626, Virtual, October 2020.

© 2020 MICCAI. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the MICCAI.

Last Modified: October 15, 2020
© 2020