Skip to main content
Once a policy is seeded and submitted, improve it with an evidence-first loop. The discipline matters more than any single change: state what you expect, gather data that would confirm or falsify it, change one thing, and verify against variance before promoting.

​
The loop

1

Define the objective

League score, beating the top N, broad robustness — the objective determines opponent selection, episode count, and the promotion threshold.
2

Run hosted evals

Use experience requests. Use pairwise evals to diagnose and broad top-N/random evals to guard against overfit. Persist every request body, episode row, replay, artifact, and log.
3

Reconstruct behavior

Join replay state (what happened) with artifact state (what your policy saw and why it acted). If the join keys are missing, add them before running a large eval.
4

Triage logs first

Inspect stdout/stderr for failed episodes. A traceback, malformed action, or provider failure is a policy bug, not evaluation noise — fix it before interpreting aggregate score.
5

Hypothesize and change one thing

Write a specific, causal, falsifiable hypothesis with an expected metric movement and a rollback condition. Make one scoped change per candidate so the eval result is attributable.
6

Verify with variance

Re-evaluate against the target, the previous champion, and broad guardrails, with enough episodes to estimate stderr. Report mean, stderr, and seat/role/opponent breakdowns.
7

Promote or reject

Promote only when the candidate improves the objective, guardrails pass, the failure rate does not increase, and the rollback ref is known. Never promote on a single pairwise win.

​
What you optimize against

Each episode produces a results file and a per-slot scores array — the game’s source of truth for how every player did. That score is the signal you optimize, and you can compare it across episodes, opponents, and seats. You can also generate your own evidence. A policy may upload a private per-slot player artifact .zip during the episode (via COWORLD_PLAYER_ARTIFACT_UPLOAD_URL), then download it afterward as a durable eval dataset — structured observations, decisions, parsed features, and fallbacks you join against the scores to explain why a policy acted. See Player artifacts.

​
Optimizing across games

Most Coworlds are unique: their mechanics differ enough that a strong policy is usually specialized for a single game, not one model that plays all of them. The transferable asset is the optimization skill — the patterns you (and your coding agents) build for reading a manifest, reconstructing behavior from replays and artifacts, and running this evidence-first loop carry from one game to the next, even when the policy code does not.

​
The eval ladder

smoke → diagnostic → candidate → guardrail → promotion. Never promote from smoke or diagnostic-only evidence. Split XP requests that are near the backend’s max size into smaller batches and aggregate them.

​
Common failure modes

  • Wrong active player before upload/submit.
  • Using a policy name ref where the API expects a policy version UUID.
  • Artifacts missing because logs were fetched without --artifact.
  • Promotion based on one pairwise win while broad leaderboard score regresses.
  • Treating a slot scoring the minimum as a strategy problem when it was a disconnect/no-show.
Sources: adapted from Metta-AI/optimizer-agent AGENTS.md, skills/coworld-operations/SKILL.md, skills/base-optimizer-framework/SKILL.md, and skills/hosted-xp-evals/SKILL.md. Game-specific eval sizing lives in the game’s own skills (for Crewrift, see Crewrift strategy).