Variations in Multi-Agent Actor–Critic Frameworks for Joint Optimizations in UAV Swarm Networks: Recent Evolution, Challenges, and Directions

Abstract

Autonomous unmanned aerial vehicle (UAV) swarm networks (UAVSNs) can efficiently perform surveillance, connectivity, computing, and energy transfer services for ground users (GUs). These missions require trajectory planning, UAV-GUs association, task offloading, next-hop selection, and resource allocation, including transmit power, bandwidth, timeslots, caching, and computing resources, to enhance network performance. Owing to the highly dynamic topology, limited resources, stringent quality of service requirements, and lack of global knowledge, optimizing network performance in UAVSNs is very intricate. To address this, an adaptive joint optimization framework is required to handle both discrete and continuous decision variables, ensuring optimal performance under various dynamic constraints. A multi-agent deep reinforcement learning-based adaptive actor–critic framework offers an effective solution by leveraging its ability to extract hidden features through agent interactions, generate hybrid actions under uncertainty, and adaptively learn with scalable generalization in dynamic conditions. This paper explores the recent evolutions of actor–critic frameworks to deal with joint optimization problems in UAVSNs by proposing a novel taxonomy based on the modifications in the internal actor– critic neural network structure. Additionally, key open research challenges are identified, and potential solutions are suggested as directions for future research in UAVSNs.