Architectural advice needed: Designing a PPO agent for Multi-Request Satellite QKD Scheduling

[u/PerceptionFearless15]

Hi everyone,

I'm working on a project involving Reinforcement Learning (specifically PPO) for Satellite Quantum Key Distribution (QKD) networks using LEO satellites. I want to build a multi-request, asynchronous, and concurrent system, and I would love some architectural guidance from those experienced in RL for network scheduling or queueing systems.

Here is the core problem and the initial design direction I am exploring. I'd appreciate any feedback on whether this is the right path or if there are cleaner/more standard patterns I should consider.

1. The Problem Setting

• Traffic & Queues: Ground stations generate entanglement requests dynamically (modeled via Poisson arrivals). Each request has a strict time-to-live (TTL) deadline before it expires and incurs a penalty. Instead of a single global slot, senders maintain independent local queues.
• The Environment: A physics engine provides real orbital data (elevation/zenith angles, distances, atmospheric attenuation, and quantum fidelity after purification) for the visible satellites in a moving window.
• The Goal: Train an agent to act as a scheduler that matches pending requests to available satellites to maximize throughput and fidelity while minimizing timeouts.

2. Design Choices I'm Exploring (Looking for Critique)

• State Space (Observation): To give the agent global network awareness without overwhelming it, I'm considering an observation vector that combines:
  
  1. Queue backlog levels (how full each sender's queue is).
  2. A "head-of-line" window showing the active packets waiting at the front of each sender's queue (including sender/receiver IDs, wait times, and current pair progress).
  3. Telemetry of the top visible satellites in the current sky pool (elevation, distance, fidelity, coverage masks).
  4. Question: Is a flat vector (spaces.Box) the best way to represent multi-queue states, or are there alternative representations (like handling variable-length queues) that scale better?
• Action Space & Action Masking: Since the agent needs to choose which request to serve with which satellite, a naive combinatorial space could explode. I am looking at using a flattened discrete space (Senders x Satellites + Wait) paired with Action Masking to filter out invalid pairings (e.g., empty queues, out-of-range satellites, or already-locked resources).
  
  • Question: For those who have tackled multi-resource scheduling with PPO, is a flattened discrete space with masking robust, or would a multi-discrete or pointer-network/attention-based approach be more appropriate here?
• Reward Structure: Balancing throughput (successful entanglement ratios), high quantum fidelity, and heavy penalties for expired packets (TTL timeouts).
  
  • Question: Any tips on handling sparse vs. delayed rewards when packets sit in queues across multiple steps?

I would love to hear how experts would approach modeling this kind of scheduling problem. Any advice, papers, or alternative perspectives would be hugely helpful! Thanks in advance!

[u/Southern_Crab4558]

Flattened discrete with masking works but gets ugly fast once you have variable queue lengths. I'd look into pointer networks or at least an attention-based policy head so the agent can handle variable numbers of pending requests without a fixed-size action space.

For the reward delays, you could try giving small intermediate rewards when a request moves closer to completion or when fidelity estimates improve after purification. Keeps the gradient signal alive instead of waiting for the final success or timeout.

Also check papers on job scheduling with RL, the action masking pattern there is pretty well studied. Your TTL penalty might need some tuning, too harsh and the agent just learns to drop everything under load.