Deepfake Detectors Are Slowly Coming Of Age, At A Time Of Dire Need

DEF CON While AI was on everyone’s lips in Las Vegas this week at the trio of security conferences in Sin City – BSides, Black Hat, AND DEF CON – there were a lot of people using the F-word too: fraud.

The plummeting cost of using AI, coupled with the increasing sophistication of deepfakes and electronic communications becoming the norm, means that we’re likely facing a massive amount of machine-learning mayhem. Deloitte estimates deepfake fraud will cost the US up to $40 billion by 2027, but everyone we’ve spoken to thinks that’s an underestimation.

Sam Altman’s comment last month that “AI has fully defeated most of the ways that people authenticate currently, other than passwords,” has ruffled some feathers in the security industry, with various vendors selling software that they claim does just that. But others are more cautious about its capabilities.

Karthik Tadinada, who spent over a decade monitoring fraud for the UK’s biggest banks when he worked at Featurespace, said that the anti-deepfake detection technology he has encountered manages about a 90 percent accuracy rate for spotting crime and eliminating false positives. That may sound good but it really shows the scale of the problem for the future.

“The economics of people generating these things versus what you can detect and deal with, well actually that 10 percent is still big enough for profit,” said Tadinada, who notes the costs of generating ID are only going to fall further.

Video impersonation predates AI, and Tadinada recounted cases where security teams had spotted fakers in high-quality silicone masks, but said that machine learning has turbocharged this. He and fellow speaker Martyn Higson, who also is ex-Featurespace, demonstrated the easy overlay of the face of British Prime Minister Keir Starmer on Higson’s body and a pretty good mimicry of his voice, all just using a MacBook Pro.

This wasn’t good enough to fool anti-scanning technology – AI tends to make the jowls more puffy and stiffen up the appearance of human faces – but it would certainly be good enough for propaganda or misinformation. This was demonstrated this week when journalist Chris Cuomo posted a deepfake video of US Representative Alexandria Ocasio-Cortez (D-NY) apparently accusing actress Sydney Sweeney of “Nazi propaganda,” before pulling it and apologizing.

Mike Raggo, the red team leader for media monitoring biz Silent Signals, agreed, pointing out that the quality of video fakes has improved drastically. But new techniques were going mainstream that might detect such fakes more easily.

He does have skin in the game. Silent Signals developed a free Python-based tool, dubbed Fake Image Forensic Examiner v1.1, for the launch of GPT-5 by OpenAI last week. This will take an uploaded video and sample frames one at a time to look for manipulation, such as blurring on the edges of objects in the video, comparing the first, last, and middle frames for background anomalies.

In addition, examining the metadata is absolutely key. Video manipulation tools, both commercial and open source, typically leave traces of code in the metadata, and a good detection engine must have the ability to perform such searches.

Paint me a picture

All three speakers agreed that images are probably the most worrying method for fraudsters to use, not only because of the ease with which they are being created, but also the increasing reliance businesses are placing on them.

Tadinada’s experiences in the banking sector made him particularly concerned about manipulated images being used in large-scale fraud. The COVID lockdown made people less willing to do banking in person and some financial institutions plunged into electronic records without considering the possibilities for fraud.

For example, to open a bank account in the UK, you’ll need to show documents such as a valid ID and a recent utility bill. Both are easily forged, as he demonstrated on stage, and can be difficult to spot electronically. Raggo agreed, particularly after visiting those announcing deepfake detection tools this week, but said there are some promising areas of deepfake spotting in evidence.

“I saw a couple of vendors at Black Hat,” Raggo told The Register. “I personally wouldn’t comment on them until I had a chance to actually test them properly but from what I saw and some of the demos that were going on, they looked interesting.”

Whichever tool people use, there are key things that it must do, Raggo argued, starting with metadata analysis. The metadata generated by AI, for example, usually lacks key code such as its International Color Consortium (ICC) profile, showing the color balance used and there is often vendor-specific metadata, such as Google’s habit of embedding “Google Inc” in the metadata of all Android images.

Edge analysis is also key, he suggested. This involves investigating the edges of objects in the picture to examine for blurring or inconsistencies in their brightness. Pixel variance – the amount by which colors in an object vary – can also be used to spot manipulation.

Going by the voice

Such techniques aren’t as useful in spotting voice deepfakes, however, and these kinds of vocal attacks are on the rise.

In May the FBI warned that a fraud campaign using AI-generated voices of US politicians was being used to trick people into handing over access to government systems from which financial information could be extracted. But the FBI’s advice to counter this was non-technical – telling users they should verify the source separately and listen for any inconsistencies in vocabulary or accent, while acknowledging that “AI-generated content has advanced to the point that it is often difficult to identify.”

It’s not the only American federal agency that is worried about the spread of voice cloning. Last year the Federal Trade Commission sponsored a year-long competition to detect AI-generated voices, but was only paying a measly $35,000 in prize money.

There are perfectly legitimate uses for such voice cloning technologies, such as for transcription, adding voice dubbing in media, and adding speech capabilities to bots in call centers. Microsoft has it as a function of Azure producing watermarked clone outputs (although these are far from perfect) and when The Register tested a voice generator from Silicon Valley startup Zyphra, the results were disturbingly good.

Such technology, however, can also be a huge help to fraudsters. If someone uses a recorded voice sample it is possible to generate a clone without the subject’s knowledge and the longer the audio sample, the better the clone is likely to be.

A study by the non-profit publication Consumer Reports into six companies offering voice cloning services found two-thirds of them made little attempt to stop the abuse of their applications – users simply had to tick a box to say that they had a legal right to clone the voice sample.

Only one of the companies tested, Resemble AI, actually required a real-time audio clip to use the service, although testers did manage to fool it some of the time with recorded audio. However, the results using this third-party audio were not as accurate due to sound issues.

Many voice-cloning businesses have also started adding deepfake detection to their product portfolios, including Resemble. CEO Zohaib Ahmed told The Register that the data cloning companies have in their servers can generate valuable fake-spotting tools.

“We’ve got a large database of real and cloned voices and comparing and contrasting them yields some valuable insights,” he said. “We’ve identified a series of artifacts, some undetectable to the human ear, that make spotting fakes easier.”

The sanity test and the rise of the GANs

As we’ve seen in the traditional security sphere, there’s no 100 percent technological fix for spotting malware and hacking, and so it is for deepfakes. Just as you can’t stop an absent-minded human from clicking on a malware-laden file, so too we can’t expect people never to be fooled by deepfakes.

“You’ve got to have a sense of precaution,” red team leader at Sophos Eric Escobar told The Register. “Verification is absolutely key, particularly if money is involved. Ask yourself ‘Is this in character?’ and then check if you’re at all uncertain.”

This is particularly relevant to the finance industry, Tadinada suggested. The use of deepfake scanning technology is all well and good, but financial transactions also need to be monitored as an indication of their use, as is the case with other types of fraud.

The use of Generative Adversarial Networks (GANs) to improve deepfakes is something that worried everyone we spoke to. GANs use two AI engines, a generator and a responder to make fake output more convincing. The generator creates media, the responder tries to spot manufactured content, and the two continue this process to improve the realism of the output.

The results aren’t perfect, and the process can currently leave tell-tale signatures in the deepfake’s metadata, Raggo noted. But the technology promises to produce ever-more realistic results and will lead to fraudsters having more success. ®

Original Source