DeepSeek Harness plugin for model controlled experiments

Project overview

dsh-bench Run one task across several models, tool surfaces, or context configurations, and get back a controlled-experiment table: which arm actually passed, what each cost, and where a slower arm's time went. When the experiment's design or its result is unsound, the report says so to your agent directly — a task every arm solves along the same trajectory measures typing speed, not capability.

The plugin registers three tools: benchrun to measure, benchhistory to compare against past runs, and benchgc to reclaim the session logs the trials leave behind. Each trial is a fresh one-shot spawn subagent that sees only the task text — no parent history — so the arms are comparable. Why

The plugin ecosystem has plenty of tools that observe a single session — usage dashboards, context breakdowns, trajectory diagnostics — and none that compare two. Without a comparison there is no way to answer whether a cheaper model, a smaller tool surface, or a shorter output budget actually changes the outcome; there are only opinions about it.