Hundreds of the nation’s smartest computer scientists gathered for a conference in Denver last fall, and Tuan Tran had a big test for them:
Could they tell the difference between the real Brad Pitt and an AI-generated version of him?
Tran, a doctoral candidate in computer science at CU Boulder, rolled the tape on a large projector. Only half of the audience could spot the deepfake.
“These deepfake generating models are becoming very strong,” Tran said.
To address this, Tran and a team of researchers at CU Boulder set out last year to build their own system for spotting fakes in video. It uses far less computing power to achieve the same results as the current detection models.
The team of scientists at CU Boulder who developed a new deepfake detection system includes, from left, Eric Keller, Sangtae Ha, Shirin Ebadi and Tuan Tran. (Credit: Scott Franz/CU Boulder)
Their findings were presented at SIGCOMM, a flagship conference in computer networking and systems held in August in Denver.
Bad actors have used AI-generated deepfakes to impersonate financial officers at major companies and convince employees to wire large amounts of money. One such deepfake scam in Hong Kong resulted in someone wiring $25 million to fraudsters.
Millions of people remain vulnerable to such trickery when they hop onto video conferencing calls. Tran said Zoom, Apple FaceTime and Google Meet lack real-time deepfake detection. Microsoft Teams does too but the company recently announced an effort to start providing it.
The commercially-available systems designed to flag imposters require a massive amount of computing power, making them hard to scale and too expensive for most users.
Tran’s team set out to determine whether deepfake detection could be simplified enough to bring to the masses.
“Some people thought this was impossible because of the system requirements and current architecture,” said Sangtae Ha, a computer science professor at CU Boulder who co-led the research team. “But by looking into the system from another perspective, you can actually solve the problem.”
Problem pixels
The team spent many hours in front of a whiteboard deep inside CU’s engineering center trying to crack the code and figure out a way to simplify deepfake detection. Sometimes they wouldn’t go home until 1 a.m.
They worked through fall break, Christmas and even the New Year.
“The experience itself was very joyful, even when it was intense,” said Shirin Ebadi, a doctoral candidate in electrical and computer engineering, and a co-author. ”The most exciting part is doing something that’s never been done.”
She said existing deepfake detectors work well, but they need to see the actual picture to work. Video travels the Internet in compressed form, so before a detector can examine a face, it has to reconstruct every frame of every participant’s video at full resolution.
That takes a lot of computing power.
The researchers discovered they didn’t need to reconstruct the video at all to spot deepfakes. The clues were already sitting in the compressed form of the video that travels over the network.
Imagine a video clip of a person speaking in front of a plain wall. Video compression doesn’t send every pixel of every frame. Instead, it describes the changes that occurred since the frame before.
The wall barely changes in the video, so it’s covered by a few large blocks of pixels, while the face and its shifting expressions are carved into many smaller blocks to capture the detail.
When someone’s appearance changes abruptly, a compressor has to write down a lot of new information. The team measured how much of that new information lands on someone’s face and how widely it’s spread.
Crucially, the method reads these numbers directly from the compressed video without ever decompressing it to a higher resolution. Only when something looks suspicious does the detector ask the sender’s device for a complete, fresh frame and use AI to check for a deepfake.
“We only had to look at data from one out of every hundred frames, instead of every frame,” Ha said.
Scaling up
The team demonstrated that the detection system could spot deepfakes using 143 times less computing power than the existing models. They tested it on a library of 1,000 online videos containing deepfakes, and reported near-perfect accuracy.
Eric Keller, an electrical and computer engineering professor at CU Boulder and a co-leader of the project, said the team wants to adapt the detection system so it can be scaled up and used for large-scale video conferencing.
The team is betting that as deepfakes become a bigger problem, the demand for new detection technology will only grow.
Source: www.colorado.edu
