Most reinforcement learning benchmarks reset the world after every episode. Real operations never reset. Skyfall AI’s MORPHEUS targets that gap. It is a persistent enterprise simulation platform for continual reinforcement learning (CRL).
What is MORPHEUS?
MORPHEUS is grounded in the Big World Hypothesis (Javed & Sutton, 2024). It says the world’s complexity exceeds any agent’s representational capacity. As a result, the environment looks non-stationary even under fixed dynamics.
To force continual learning, MORPHEUS requires three properties: persistence, non-stationarity, and operational complexity. Persistence means past decisions compound into future dynamics. Non-stationarity means any fixed policy eventually becomes suboptimal. Operational complexity means no fixed optimal policy exists.
Each environment is a self-contained TypeScript world plugin. It exports Operational Descriptors (ODs), a simulation scheduler, seed data, and documentation. An OD defines the step-by-step execution plan for a capability. Agents act through a capability API, and each call triggers an OD execution.
How the Platform Works?
Building on that architecture, non-stationarity comes from two engines. First, a failure injection engine inserts typed disruptions between OD steps. It draws from eleven failure types, including missing_data, dependency_failure, and rate_limit. It runs at four preset rates: light (5%), realistic (8%), moderate (15%), and aggressive (30%).
Second, an asynchronous configuration shift controller changes failure presets and demand at fixed timestamps. It runs independently of the training loop, so shifts never align with gradient updates. This stops the agent from using update periodicity as a proxy clock.
Alongside these engines, reward comes from three operational verifiers logged natively by the platform. These are failure event signals, financial ledger status, and resource throughput. The composite reward combines them. Default weights are w_f = 0.5 and w_l = w_p = 0.25.
# Composite reward — MORPHEUS, Appendix C (default weights).
def clip(x, lo, hi):
return max(lo, min(hi, x))
def composite_reward(tickets, actual_cost, planned_cost, units, capacity,
w_f=0.5, w_l=0.25, w_p=0.25):
r_f = -sum(t["severity"] for t in tickets) # failure event signal
r_l = clip(1 - actual_cost / planned_cost, -1, 1) # financial ledger
r_p = clip(units / capacity, 0, 1) # resource throughput
return w_f * r_f + w_l * r_l + w_p * r_pUnder the upper-bound assumptions (zero failures, minimum cost, full throughput), the bound per configuration equals 0.50.
Policy Initialisation
Because the action space is large, pure RL from scratch is impractical. Therefore MORPHEUS uses a two-stage pipeline. A frontier model (Gemini 3.1 pro) collects trajectories using the ReAct framework. These traces then fine-tune Qwen3-14B via supervised fine-tuning (SFT).
Consequently, every RL run starts from this shared SFT checkpoint. This isolates continual learning behaviour from basic operational competence. All baselines then use PPO as the base optimizer for online post-training.
The Six-Metric Evaluation Protocol
With training defined, cumulative reward alone is not enough. A scalar sum hides performance across a non-stationary horizon. So the research team propose six metrics instead. These are per-configuration reward, adaptation speed, forgetting, recovery time, stability, and performance gap.
Among these, adaptation speed is the headline metric. It counts steps until the running-average reward reaches half the upper bound. Two supplementary diagnostics also track relative adaptation advantage (RAA) and plasticity via effective rank.
Baseline Results
Using this protocol, the research team tests four algorithm families from the shared SFT checkpoint. Two tasks are defined. Task 1 is dynamic resource allocation under structured drift. Task 2 is scheduling under drift with delayed effects.
| Family | Mechanism | Outbound Task 1 | Outbound Task 2 | Inbound Task 2 |
|---|---|---|---|---|
| PPO | No CL mechanism | Failure baseline | Adapts only early | Baseline reward |
| HER | Hindsight replay | Mid reward | Best reward | Best reward, top rank |
| EWC | Weight consolidation | Best reward | Best adaptation | Weakest reward |
| LCM | Latent context model | Fastest adaptation | No advantage | Best adaptation |
Across these results, no single family dominates. On process-outbound Task 1, EWC leads reward and LCM adapts fastest. On Task 2, HER leads reward while LCM loses its edge under delayed reward. Meanwhile, mean performance gaps sit near 1.0 for every method. That signals a large settled-state deficit, not a minor tuning gap.
Notably, PPO and HER generally adapt only in the first configuration. They then fail to adapt in later regimes, even without label signals.
Use Cases with Examples
In practice, MORPHEUS suits several reader roles. For AI engineers, it tests whether an agent detects regime shifts without labels. For example, demand switches from low to bursty, and the policy must adapt with no signal.
For data scientists, it stresses delayed credit assignment. For example, On-Time In-Full (OTIF) delivery is observable only days after the dispatch decision. For software engineers, the TypeScript plugin format allows swapping rewards or toggling observability without changing dynamics.
Strengths and Weaknesses
Strengths:
- Persistent worlds with no resets, matching deployed enterprise systems.
- Parameterisable, reproducible regime shifts for fair cross-algorithm comparison.
- Rewards from native operational verifiers, needing no external annotation.
- Open-sourced evaluation code (
Skyfall-Research/morpheus-evals).
Weaknesses:
- Only two of five environments are evaluated so far.
- The upper bound assumes zero failures, so it stays optimistic.
- Shifts are externally triggered, not driven by compounding decisions.
- Reward weights are research variables, not validated industry objectives.
Key Takeaways
- MORPHEUS runs persistent enterprise worlds that never reset, unlike episodic RL benchmarks.
- It ships five environments; two are evaluated here:
process-outboundandprocess-inbound. - A six-metric protocol scores per-configuration reward, adaptation, forgetting, recovery, stability, and gap-to-upper-bound.
- Four baselines (PPO, HER, EWC, LCM) all sit far below the theoretical upper bound.
- No single algorithm wins; reward and adaptation speed pick different winners.
Check out the Paper and Project Page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Credit: Source link
