Detection of Deepfake using computer vision and deep learning

Abstract

In this era of Metaverse and technological advancement, Deepfakes are one of the most alarming concepts. Deepfakes are mostly synthetically-generated manipulated photos or videos which is created swapping a face of a person using Deep learning, Generated Adversarial Network (GAN), autoencoder-decoder pairing structure. There are several other Deepfaking tools such as; FaceSwap, DeepFaceLab, DFaker, DeepFake-tensorflow etc. Using Generative Adversarial Network (GAN), Deepfakes have become smoother and more realistic in making the fake videos. DeepFakes can become concerning if it is used for political purpose, committing fraud, spreading misinformation, pornography, defamation on social media etc. This possesses a security threat on people’s lives knowingly or unknowingly. As a result, it is visible that DeepFakes can be very distressing on the wrong hand if not detected properly. Our purpose is to detect the DeepFake videos as successfully as possible. We want to focus on detection using Deep learning approaches also using Image recognition and computer vision. For this detection, we used a dataset of videos, which included both real and fake videos. We have successfully extracted it from Kaggle where we have found dataset of more than 2352 videos from DeepFake Detection Challenge(DFDC) and FaceForensics++. To detect the fake videos, we followed the method of employing temporal feature and exploring visual artifacts within frames. Employing temporal feature uses LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network) whereas visual artifacts within frames mostly employs deep learning method to detect DeepFakes. We ensembled LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network) to detect DeepFakes successfully. ResNeXt101_32x8d have been used to extract features and a custom CNN model is added with LSTM for better accuracy for detecting DeepFake. The accuracy of the model was 94.05%. After further improvement and with the introduction of learning rate schedulers, we were able to achieve better accuracy. We have used CosineAnnealingLR, CyclicLR, MultiStepLR and ReduceLRonPlateau among which MultiStepLR gave the highest accuracy of 95.33%.