Fake data is everywhere now. Fake person in videos, cloned voices on a call, fake images used to trick people. Anyone, literally anyone, can make this stuff for free with basically no tech knowledge. So, we built a system to catch it. In this post, you will explore how we built a decent Deepfake detector with almost no compute resources. You can visit the GitHub repo for this here: Deepfake Detection Suite
Deepfake is a rising concern, it causes harm to innocent people. People use them to:
Most decision tools only check one type of media. Video only, or Audio only. A very few tools cover everything, but require heavy compute power to run or are simply very costly. In short, normal people cannot use them.
We wanted one tool. Checks video, audio, and images. Runs fast. Works on normal machines. That is the whole goal. So, we just built one!
One web app. One upload button. Three different functionalities.
| Type | Accuracy |
|---|---|
| Video | about 89% |
| Image | about 89% |
| Audio | about 98% |
Build with TensorFlow and Keras, trained on Colab with one T4 GPU and 12GB RAM for the initial version. The current version was trained on a Ryzen 3 5300U CPU with 8GB RAM. Very resource efficient!
Imagine a guy holding a tiny magnifying glass. He scans a photo in small patches. One patch at a time. He looks for edges, textures, weird blending, odd lighting. Do that scan many times, stack the results, and the model builds up a full picture of what looks fake and what looks real.
That is basically a CNN. Convolutional Neural Network. It slides small filters(called Kernel) across an image and learns patterns.
Example: show a CNN 1000 cat photos and 1000 dog photos. It learns cat ears look pointy, dog ears look floppy. Show it a new photo, it checks for those shapes. Same here, but instead of cat vs dog, it learns real face vs fake face.
But normal CNNs are slow. They mix space and colour info together in one heavy step for every filter. So, to speed up this, we decided to use Xception.
Xception is smarter. It splits the heavy steps(discussed above) into two light steps:
Two light steps beat one heavy step. Same learning power but much faster. This enables Xception to run in near real time and for the same reason, we decided to go with it.
We start with ImageNet pretrained weights (a huge head start from millions of general images) and then fine tune on Celeb-DF, a dataset full of real and fake celebrity videos.
n frames from it, spread evenly across the videon scores togetherCNNs mostly judge frames one at a time. They catch weird pixels or textures inside a single frame really well. But they are not great at noticing that something is off across frames over time, like unnatural blinking patterns or jittery motion. That is a gap which we aim to fix.
Honestly, this one is simple to explain because it just reuses the video model.
An image goes through the exact same pipeline as video, except there is only one frame instead of
n. Face detection, resize to 128x128, normalize, run through Xception, get a score, apply the
0.5 cutoff.
It works okay because a still image is basically just one video frame. But it was never trained specifically on image-only fake datasets. It is a stand-in solution until a dedicated image model gets built later using datasets like FaceForensics++. We also think of building our own dataset but with current resources, that does not seem feasible.
A cloned voice can sound pretty convincing for a single word. But over a full sentence, things start slipping. Pitch drifts oddly. Rhythm gets a little off. Words start to overlap. Breathing pauses land in weird spots. These are timing problems, not pixel problems. You need a model that remembers what happened earlier in the clip to catch this.
LSTM or Long Short-Term Memory, is a special kind of RNN, a network built to process sequences step by step while keeping a memory of what came before.
Plain RNNs are bad at long memory. Info fades out fast the further back it goes, this is called the vanishing gradient problem. LSTM fixes this with gates. Think of gates like small decision switches:
These gates let the network hold onto useful signal from far back in the clip without losing it. Perfect for catching subtle, slow-building weirdness in a fake voice.
Example: Assume you listen to a 10 second voice clip. A normal RNN forgets how the sentence started by the time it ends. LSTM remembers the start, middle, and end together, like a person actually listening to the whole sentence.
We use something called MFCC features to study audio. MFCC stands for Mel-Frequency Cepstral Coefficients, basically a compressed shape of how the human voice sounds, standard in almost all speech tasks. Example: like turning a voice into a fingerprint. Two different voices leave two different fingerprints, even saying the same word.
This data is scaled, broken into windows, so that the model reads it as a sequence. These windows are fed through the LSTM model, the windows' prediction becomes the final answer.
The research also looked at Transformer models. Vision Transformers (ViT) and TimeSformer for video. Wav2Vec 2.0 and SpeechT5 for audio. These models use something called self-attention, which lets them look at relationships across an entire input at once, not just nearby patches or nearby time steps like CNNs and RNNs do.
On paper, Transformers beat CNNs and RNNs on many benchmarks. They are genuinely stronger.
Example: CNN reads a page word by word in order. Transformer reads the whole page at once and connects word 1 to word 50 instantly. Powerful, but needs a much bigger brain to do that.
But they come with a big cost. They need massive amounts of training data and heavy compute power. Our whole goal was a lightweight system that could eventually run on a mobile phone through TFLite. Transformers do not fit that goal right now, not without serious compression work. So we left this as a future upgrade path once lighter Transformer variants become more practical.
Being honest about the gaps:
This project combines two very different tools for two very different problems. CNNs are great at spotting weird pixels in a single frame, so they handle video and image checks. LSTMs are great at spotting weird timing across a sequence, so they handle audio checks. Both get squeezed down small enough to run on regular hardware instead of a massive server farm.
It is not a finished, perfect system. The image model needs real training of its own. But it proves something important: solid deepfake detection does not have to live only in research labs or big tech companies. It can run on a laptop, and eventually a phone, right where people actually encounter fake content in the first place.
That's it for now, if you have any query, feel free to drop me an email at aanis@clipb.in.
Thanks for Reading,
Cheers!
> Go Back