Is it performing?
Production readiness should not rest on a probabilistic “single good run”; pilot and production share one measurement scale, with aligned acceptance criteria.
Freyja Eval
From pilot to production, with confidence
Transferable eval design abstracted from Freyja — narrowing the trust gap
Production readiness should not rest on a probabilistic “single good run”; pilot and production share one measurement scale, with aligned acceptance criteria.
Point logs are insufficient to characterize multi-agent failures; bad cases should be classifiable with drill-down to root cause, feeding back into workflow and architecture.
Pre-release gates block regression; online monitoring and graded degradation provide fallback; fixes are codified as retestable samples; critical checks hold veto over release.
Dual-trace guardrails × deterministic outcome verification × dual-track loop—the general methodology instantiated on Freyja
Pilot evidence: selected cases—detected, attributed, and codified
the supervisor relayed configuration context to subagents as Markdown documents, and identical inputs would probabilistically drop key parameters across runs—every agent looked correct in isolation; the failure lived at the handoff boundary. Judgment: the problem was the translation layer, not model capability—so instead of larger prompts, handoffs moved to a structured requirements brief (JSON), with a planned upgrade to versioned shared-state reads; a green single run no longer counts as passing without repeated chain-level verification, and the failure is codified as a regression case.
after multi-turn, multi-file uploads, the aggregated requirements brief missed items relative to source materials; planning, execution, and self-check all validated against this incomplete brief—everything looked green, yet the platform configuration was still missing items. Judgment: agent self-reports and internal self-checks cannot be trusted—the product shipped incremental multi-turn merging; the eval side added layered brief defenses (schema, source fidelity, user confirmation, drift prevention) with platform-level outcome verification as backstop; upload persistence is on the roadmap.