Collaborative cross knowledge distillation between self-supervised convolution and spiking neural network

Abstract

Spiking Neural Networks (SNNs) are energy efficient, but they have a high inference latency and historically have lower accuracy than ANNs. However, the performance of SNNs often lags behind conventional neural networks like Convolutional Neural Networks (CNNs), particularly when it comes to complex datasets and real-time applications. Hybrid approaches combining CNNs and SNNs aim to leverage the strengths of both, but they typically struggle with slow convergence, high latency, and challenges in effective knowledge transfer between the two models. To address this, we present a highly optimized Teacher–Student (CNN to SNN) Knowledge Distillation (KD) pipeline that uses a multi-signal strategy designed for low-latency spiking dynamics. Our KD framework uses rate-based feature KD, which uses cosine similarity to effectively bridge the modality gap between the networks, and per-timestep logit alignment to stabilize temporal dynamics. This pipeline achieves competitive accuracy at low latency (T < 4), supported by learnable LIF dynamics and a spike-rate regularizer for sparsity. A highly effective route for implementing reliable SNNs in real-time, resource-constrained environments is established by this optimized methodology, which also exhibits up to 14 times faster convergence. To further enhance the learning process, we employ the BYOL self-supervised approach to train the CNN teacher, and integrate an attention mechanism in the SNN student for efficient knowledge distillation. Our experimental results show that, on the MNIST dataset, the CNN teacher achieves 99.43% accuracy, with the SNN student achieving 96.00%. On CIFAR-10, the CNN teacher reaches 85.27%, and the SNN student achieves 82.56%, on Imagenette, the CNN teacher achieves 80.44% accuracy, with the SNN student achieving 78.23% demonstrating the effectiveness of our collaborative distillation framework. This work paves the way for more efficient, real-time implementations of SNNs in low-latency applications, with significant improvements in accuracy and convergence.