Deep Reinforcement Learning Based Resource Allocation in Cooperative UAV-Assisted Wireless Networks
Abstract : We consider the downlink of an unmanned aerial vehicle (UAV) assisted cellular network consisting of multiple cooperative UAVs, whose operations are coordinated by a central ground controller using wireless fronthaul links, to serve multiple ground user equipments (UEs). A problem of jointly designing UAVs’ positions, transmit beamforming, as well as UAV-UE association is formulated in the form of mixed integer nonlinear programming (MINLP) to maximize the sum UEs’ achievable rate subject to limited fronthaul capacity constraints. Solving the considered problem is hard owing to its non-convexity and the unavailability of channel state information (CSI) due to the movement of UAVs. To tackle these effects, we propose a novel algorithm comprising of two distinguishing features: (i) exploiting a deep Q-learning approach to tackle the issue of CSI unavailability for determining UAVs’ positions, (ii) developing a difference of convex algorithm (DCA) to efficiently solve for the UAV’s transmit beamforming and UAV-UE association. The proposed algorithm recursively solves the problem of interest until convergence, where each recursion executes two steps. In the first step, the deep Q-learning (DQL) algorithm allows UAVs to learn the overall network state and account for the joint movement of all UAVs to adapt their locations. In the second step, given the determined UAVs’ positions from the DQL algorithm, the DCA iteratively solves a convex approximate subproblem of the original non-convex MINLP problem with the updated parameters, where the problem’s variables are transmit beamforming and UAV-UE association. Numerical results show that our design outperforms the existing algorithms in terms of algorithmic convergence and network performance with a gain of up to 70.
? The existing deep reinforcement learning based methodd only handle UAVs’ 2D flight and simple transmission control decisions.
? However, most existing research works neglected adjusting UAVs’ height to obtain better quality of links by avoiding various obstructions or non-line-ofsight (NLoS) links.
? Most existing works assumed that the ground terminals are stationary; whereas in reality, some ground terminals move with certain patterns, e.g., vehicles move under the control of traffic lights.
? Existing works obtain the near-optimal strategies in the 2D flight scenario when users are stationary, however, they are not capable of solving our target problem since the UAV adjusts its 3D position and vehicles move with their patterns under the control of traffic lights.
? A problem of jointly designing UAV0 s location, transmit beamforming, as well as UAV-user association is formulated in the form of mixed integer nonlinear programming (MINLP) to maximize the sum user achievable rate while considering the constraints of limited fronthaul capacity.
? However, these work assumed predetermined CSI as input to the optimization problem to solve for the UAVs positions and resource allocation, which is not practical.
? On the other hand, the deep reinforcement learning (DRL) approach has recently been exploited and applied to the problem of UAV position and resource allocation in the UANs.
• In this paper, we propose deep reinforcement learning based algorithms to maximize the total throughput of UAV-tovehicle communications, which jointly adjusts the UAV’s 3D flight and transmission control by learning through interacting with the environment.
• We propose three solutions with different control objectives to jointly adjust the UAV’s 3D flight and transmission control.
• we provide extensive simulation results to demonstrate the effectiveness of the proposed solutions compared with two baseline schemes.
• We pre-trained the proposed solutions using servers, and we hope the UAV trains the neural netwroks in the future if light and low energy consumption GPUs are applied at the edge.
? We evaluate the performance of our proposed scheme with DQL algorithm.
? We compare the performance of our proposed scheme and algorithm to other related schemes.
? We investigated the design of UAVs positions and resource allocation in the downlink of an UAN where cooperative UAVs scheme is considered to enhance the system performance
? ESN algorithm using multiagent Q-learning was used to predict the future positions of UEs and determine the positions of UAVs
? The outcome of this DC algorithm is then used to construct the decision policy of DQL algorithm to recompute the UAV position and this process is iterated until convergence.
|