This paper studies how a single designer or “leader” can nudge a group of independent decision-makers toward a desired outcome by paying small rewards, while trying to keep the payment cost low. Each decision-maker, called a follower, plans its actions by solving a Markov decision process (MDP). An MDP is a simple formal model for planning: it has states, actions, rules that say how actions move you between states, and rewards that the follower tries to maximize over time.

The authors model the interaction as a Stackelberg, or leader–follower, game cast as a bilevel optimization problem. At the lower level, each follower independently finds a policy that maximizes its own expected return under the reward function that includes any side payment from the leader. At the upper level, the leader chooses those side payments to maximize its own objective, which depends on the collection of followers’ best-response policies and also pays a cost for the payments themselves.

Two technical difficulties make this hard. First, the leader’s objective is generally non-concave, so standard optimization guarantees fail. Second, each follower’s MDP can have multiple optimal policies, so the leader cannot assume a unique follower response. To address these issues the authors reformulate the bilevel problem as a constrained optimization problem and develop a first-order algorithm (a method that uses gradients but does not require expensive Hessian matrix inverses). They rely on smoothness properties of MDP value functions and treat two common ways of parameterizing policies: direct parameters and softmax parameters. The paper proves that the algorithm converges to a stationary point of the original bilevel problem under the stated assumptions, and it shows how the required gradients can be computed or estimated using standard reinforcement learning (RL) techniques.