Feb 10, 2026
Launching our first simulation benchmark suite
A benchmark set for testing whether energy-system agents remain stable across perturbed demand, supply, and policy assumptions.
Benchmarks For Stability
Energy-system simulations can look convincing while hiding brittle assumptions. The benchmark suite is designed to expose that brittleness by perturbing demand traces, equipment behavior, policy constraints, and available telemetry.
We evaluate agents on performance, stability, and explanation quality. A strong run should not only reach a useful operating point; it should also make clear which constraints shaped the result and which assumptions it relied on.
The Test Families
The first family covers dispatch: storage, thermal load, flexible demand, and local generation. The second family covers planning: where to place sensors, which assets to upgrade, and when a simulated intervention becomes robust enough for a field pilot.
The third family covers research automation. It asks whether an agent can design a useful follow-up experiment after a failed run. This matters because real laboratories do not only need optimizers. They need systems that recover intelligently when a hypothesis breaks.
Failure Modes We Want
We intentionally include bad forecasts, missing sensors, nonstationary demand, policy changes, and equipment constraints that conflict with the objective. These are not edge cases. They are the conditions under which energy software proves whether it is useful.
Each failure mode is tagged with the evidence needed to diagnose it. If a controller fails because the thermal model is wrong, the benchmark should say that. If it fails because the objective was underspecified, the benchmark should say that too.
How Results Are Reported
The benchmark reports distributions, not only averages. Median performance is useful, but tail behavior is where deployment risk lives. A policy that occasionally creates a large peak can be worse than a policy with lower average savings and tighter bounds.
We also require trace-level explanations. Agents must provide enough context for a researcher to reproduce the decision path. This keeps the benchmark aligned with applied lab work instead of turning it into another opaque score chase.