Jared Cohen
Prepare to have your mind blown. Researchers have discovered a mind-boggling way to extract audio from still images and soundless videos. And guess what? It was all inspired by a sci-fi TV show called Fringe. How cool is that?
In the show, the FBI manages to extract recorded sound from a melted pane of glass. Now, some might call this a “ridiculous pseudo-science technique,” but hold on a second. Professor Kevin Fu from Northeastern University’s electrical and computer engineering and computer science department saw this review and decided to prove them wrong. He set out to show that extracting audio from images and silent videos is not only possible but also mind-blowingly awesome.
So, how does this sorcery work? Well, it turns out that cameras, while focused on capturing visuals, unintentionally pick up audio information too. Most camera phones have image stabilization technology, which uses springs to hold the camera lens in place. These springs vibrate ever so slightly when there’s a noise near the lens, causing the light to bend. It’s not something you’d notice unless you’re actively looking for it. But wait, there’s more!
Modern phone cameras have another nifty feature that amplifies this audio capture. Instead of scanning all pixels of an image simultaneously, they do it row by row. This means that the frequency information can be amplified by over a thousand times, giving us the granularity needed to extract audio.
Using this incredible information, the team developed a machine-learning algorithm named Side Eye. With Side Eye, they were able to extract muffled audio from any photo that contains light. And guess what? It actually works!
They tested their system on 10 different smartphones and achieved impressive results. The algorithm could recognize spoken digits with 80.66 percent accuracy, identify speakers with 91.28 percent accuracy, and even guess the gender of speakers with 99.67 percent accuracy. Mind-blowing, right?
Now, before you start panicking about the potential cybersecurity nightmare this could create, the team has already thought about that. They’re exploring solutions like stronger springs, locking lenses, and randomizing how the rolling shutter captures pixels. Safety first!
But here’s the really exciting part. The team believes that extracted audio could be a game-changer in legal cases. Imagine having an alibi and being able to prove it in court using authenticated video with a known timestamp. If the person’s voice is heard, they’re more than likely there. Talk about a powerful tool!
The study, which is blowing minds left and right, is available on the pre-print server arXiv. It was also presented at the 2023 IEEE Symposium on Security and Privacy. Brace yourselves, folks. The future is here!
