Multi-Agent Deep Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks with Imperfect Channels
Abstract : This paper investigates a futuristic spectrum sharing paradigm for heterogeneous wireless networks with imperfect channels. In the heterogeneous networks, multiple wireless networks adopt different medium access control (MAC) protocols to share a common wireless spectrum and each network is unaware of the MACs of others. This paper aims to design a distributed deep reinforcement learning (DRL) based MAC protocol for a particular network, and the objective of this network is to achieve a global a-fairness objective. In the conventional DRL framework, feedback/reward given to the agent is always correctly received, so that the agent can optimize its strategy based on the received reward. In our wireless application where the channels are noisy, the feedback/reward (i.e., the ACK packet) may be lost due to channel noise and interference. Without correct feedback, the agent (i.e., the network user) may fail to find a good solution. Moreover, in the distributed protocol, each agent makes decisions on its own. It is a challenge to guarantee that the multiple agents will make coherent decisions and work together to achieve the same objective, particularly in the face of imperfect feedback channels. To tackle the challenge, we put forth (i) a feedback recovery mechanism to recover missing feedback information, and (ii) a two-stage action selection mechanism to aid coherent decision making to reduce transmission collisions among the agents. Extensive simulation results demonstrate the effectiveness of these two mechanisms. Last but not least, we believe that the feedback recovery mechanism and the two-stage action selection mechanism can also be used in general distributed multi-agent reinforcement learning problems in which feedback information on rewards can be corrupted.
? In contrast with existing studies, which quantized the continuous set into discrete space, we propose utilizing the parameterized deep Q-network (P-DQN) to handle the problem with a hybrid action space composed of discrete user association and continuous power allocation.
? In this paper, we have studied the joint problem of user association and power allocation using P-DQN in the downlink of a two-tier HetNet without knowledge of the environment transition probability.
? Each SBS allocates the power to its serving UEs based on the policy determined by the P-DQN. Each UE is randomly associated with one SBS in such a way that the random association policy obeys the backhaul link constraint.
? We demonstrate that our proposed feedback recovery mechanism can effectively solve the problem of imperfect feedback channels.
? Therefore, in our design, we adopt the multidimensional DRL framework in to solve this problem.
? This paper adopts a distributed DLMA algorithm to solve the imperfect channel problem.
? we believe that the feedback recovery mechanism and the two-stage action selection mechanism proposed in this paper can also be used in general distributed multi-agent reinforcement learning problems in which feedback information on rewards can be corrupted.
• This per user power constraint can be facilitated by the actor-parameter network in the proposed P-DQN in a much easier way.
• To overcome this difficulty, in this paper, we propose employing the P-DQN for the joint power allocation and user association because of its capability of solving problems with hybrid action space.
• The user association policy is accomplished by the proposed P-DQN, whereas each SBS serves the UEs in its cluster with random powers under the total power and backhaul capacity constraint.
• It shows the convergence of the user association and power allocation algorithm using the proposed P-DQN.
? This paper puts forth an intelligent MAC protocol for a particular network—among the networks sharing the spectrum—to achieve efficient and equitable spectrum sharing among all networks.
? We develop a distributed DRL based MAC protocol for efficient and equitable spectrum sharing in heterogeneous wireless networks with imperfect channels.
? We first study the performance of DLMA in reducing the detrimental effect of the imperfect feedback channels when the objective of the agent is to maximize sum throughput.
? We then study the performance of DLMA in reducing the detrimental effect of the imperfect feedback channels when the DLMA agent aims to achieve a different objective.
|