Confessions of a Masked MDP
This paper studies the problem of learning an optimal policy and the action mask in a linear MDP model.
This paper introduces a new framework for learning an optimal policy and the action mask in a linear MDP model, which is a novel approach compared to prior work.
Keywords
Before reading this…
Applications
- →Transfer learning
To understand this paper, make sure you know these concepts first:
- Reinforcement learningfind papers →
- Markov decision processesfind papers →
Abstract
More Like ThisIn reinforcement learning, one presumes the action set to be known ahead of time, and, thus, one must know if any state-action pairs are not permissible ahead of computations. In some applications, determining this action mask is the main computational bottleneck. Isolating this problem to highlight its core properties, we study how to combine the search for an optimal policy with the learning of the mask itself, under a linear MDP model in which each attempted action reveals the exact value of a linear score determining its validity. In this framework, we first construct an algorithm based on deterministic optimistic optimization and show that it is minimax optimal, which we show is also computationally efficient. However, it is sub-optimal on non-degenerate instances, as we show. We propose an alternative, which is also computationally efficient, which plans its exploration strategy. This algorithm admits a formal interpretation as an analogue of information-directed-sampling for deterministic feedback, which may be of independent interest. Finally, we study an application of this framework to transfer learning, where the invalid action set changes over time.