EvMic: Event-based Non-contact sound recovery from effective spatial-temporal modeling
Hao Yin, Shi Guo, Xu Jia, Xudong XU, Lu Zhang, Si Liu, Dong Wang, Huchuan Lu, Tianfan Xue
2025-04-07
Summary
This paper talks about EvMic, a smart system that uses special cameras to 'see' sound vibrations on objects and turn them back into audio, like reading sound waves from a shaking bag of chips.
What's the problem?
Old methods struggle to capture high-pitched sounds or need complex setups, making it hard to record sounds from far away or without touching objects.
What's the solution?
EvMic uses laser-enhanced cameras that detect tiny vibrations, then smart software analyzes the vibration patterns to rebuild the original sound, even training on computer-made examples first.
Why it matters?
This lets people record sounds secretly from far away (like in nature documentaries) or in places where microphones can't go, helping scientists and security teams.
Abstract
When sound waves hit an object, they induce vibrations that produce high-frequency and subtle visual changes, which can be used for recovering the sound. Early studies always encounter trade-offs related to sampling rate, bandwidth, field of view, and the simplicity of the optical path. Recent advances in event camera hardware show good potential for its application in visual sound recovery, because of its superior ability in capturing high-frequency signals. However, existing event-based vibration recovery methods are still sub-optimal for sound recovery. In this work, we propose a novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal information from the event stream. We first generate a large training set using a novel simulation pipeline. Then we designed a network that leverages the sparsity of events to capture spatial information and uses Mamba to model long-term temporal information. Lastly, we train a spatial aggregation block to aggregate information from different locations to further improve signal quality. To capture event signals caused by sound waves, we also designed an imaging system using a laser matrix to enhance the gradient and collected multiple data sequences for testing. Experimental results on synthetic and real-world data demonstrate the effectiveness of our method.