Publications
Multimodal dialog based speech and facial biomarkers capture differential disease progression rates for ALS remote patient monitoring,
M. Neumann, O. Roesler, J. Liscombe, H. Kothare, D. Suendermann-Oeft, J. D. Berry, E. Fraenkel, R. Norel, A. Anvar, I. Navar, A. V. Sherman, J. R. Green and V. Ramanarayanan (2021).
In Proc. of: The 32nd International Symposium on Amyotrophic Lateral Sclerosis and Motor Neuron Disease, Virtual, December 2021.
Objective
Identify audiovisual speech markers that are responsive to clinical progression of Amyotrophic Lateral Sclerosis (ALS).
Lessons learned from a large-scale audio-visual remote data collection for Amyotrophic Lateral Sclerosis research.
Vikram Ramanarayanan, Michael Neumann , Aria Anvar, Oliver Roesler , Jackson Liscombe , Hardik Kothare , David Suendermann-Oeft , James D. Berry , Ernest Fraenkel , Raquel Norel , Alexander V. Sherman, Jordan R. Green and Indu Navar
Modality.AI, MGH Institute of Health Professions, Massachusetts Institute of Technology, IBM Thomas J. Watson Research Center, EverythingALS, Peter Cohen Foundation, Harvard University, University of California, San Francisco
Investigating the Utility of Multimodal Conversational Technology and Audiovisual Analytic Measures for the Assessment and Monitoring of Amyotrophic Lateral Sclerosis at Scale.
M. Neumann, O. Roesler, J. Liscombe, H. Kothare, D. Suendermann-Oeft, D. Pautler, I. Navar, A. Anvar, J. Kumm, R. Norel, E. Fraenkel, A. Sherman, J. Berry, G. Pattee, J. Wang, J. Green, V. Ramanarayanan: Investigating the Utility of Multimodal Conversational Technology and Audiovisual Analytic Measures for the Assessment and Monitoring of Amyotrophic Lateral Sclerosis at Scale . Accepted at Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czech Republic, August - September 2021
Accepted at Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czech Republic, August - September 2021.
Abstract
We investigate the utility of audiovisual dialog systems combined with speech and video analytics for real-time remote monitoring of depression at scale in uncontrolled environment settings. We collected audiovisual conversational data from participants who interacted with a cloud-based multimodal dialog system, and automatically extracted a large set of speech and vision metrics based on the rich existing literature of laboratory studies. We report on the efficacy of various audio and video metrics in differentiating people with mild, moderate and severe depression, and discuss the implications of these results for the deployment of such technologies in real-world neurological diagnosis and monitoring applications.
Towards A Large-Scale Audio-Visual Corpus for Research on Amyotrophic Lateral Sclerosis
A. Anvar, D. Suendermann-Oeft, D. Pautler, V. Ramanarayanan, J. Kumm, J. Berry, R. Norel, E. Fraenkel, and I. Navar: Towards A Large-Scale Audio-Visual Corpus for Research on Amyotrophic Lateral Sclerosis. In Proc. of AAN 2021, 73th Annual Meeting of the American Academy of Neurology, Virtual, April 2021.
In Proc. of AAN 2021, 73th Annual Meeting of the American Academy of Neurology, Virtual, April 2021
Objective
This presentation describes the creation of a large, open data platform, comprising speech and video recordings of people with ALS and healthy volunteers. Each participant is interviewed by Modality.AI’s virtual agent, emulating the role of a neurologist or speech pathologist walking them through speaking exercises [Fig 1] The collected data is made available to the academic and research community to foster acceleration of the development of biomarkers, diagnostics, therapies, and fundamental scientific understanding of ALS.
