Evidence and evaluation
How to evaluate Sakana Fugu beyond the benchmark chart
The official scores are useful vendor evidence. They are not a purchase order. This playbook turns the chart into a controlled comparison that measures correctness, cost, latency, failure recovery, observability, and governance on work your team actually performs.
1. Read the published table as a map of strengths
The current official Models page reports the following selected scores. They are reproduced here to compare Fugu with Ultra, not to imply independent verification by this site.
| Benchmark | Fugu | Fugu Ultra | Higher reported Fugu model |
|---|---|---|---|
| SWE Bench Pro | 59.0 | 73.7 | Ultra |
| Terminal Bench 2.1 | 80.2 | 82.1 | Ultra |
| LiveCodeBench | 92.9 | 93.2 | Ultra |
| Humanity's Last Exam | 47.2 | 50.0 | Ultra |
| GPQA Diamond | 95.5 | 95.5 | Tie |
| SciCode | 60.1 | 58.7 | Fugu |
| Long Context Reasoning | 74.7 | 73.3 | Fugu |
| MRCRv2 | 86.6 | 93.6 | Ultra |
The pattern matters more than the marketing shorthand. Ultra is ahead on many difficult suites, but Fugu ties or leads on some science and long-context entries. “Always use Ultra” is not supported even by the vendor's own table.
2. Ask what each benchmark is standing in for
Repository issue resolution
SWE Bench style tasks can inform coding-agent evaluation, but your language mix, repository size, test quality, and tool permissions may be different.
Terminal operation
Terminal benchmarks probe multi-step action and recovery. They do not automatically cover your deployment permissions, network limits, or destructive-action safeguards.
Standalone code generation
LiveCodeBench is useful for code reasoning under a defined harness. It says less about navigating a large private codebase or maintaining a long editing session.
Scientific reasoning
GPQA and SciCode can indicate difficult reasoning ability. They do not prove the model will cite internal evidence correctly or respect domain-specific review standards.
Long context
Long-context results are a starting signal. Real retrieval quality depends on where evidence appears, distractors, tool calls, and whether the output can point back to the source.
Agentic orchestration
A benchmark can reward a final answer while hiding extra latency, retries, route instability, or token use. Production evaluation must add those missing dimensions.
3. Keep the evidence boundary visible
The official product page says some baseline values are provider reported. The technical report describes the authors' training and evaluation setup. Those are valuable primary sources, but the producer and evaluator are still closely related.
Evidence you have
- Named benchmarks and scores
- A technical report with method detail
- Documented model behavior and API fields
- Qualitative vendor case studies
Evidence you still need
- Your exact prompts and tools
- Repeated runs, not one showcase
- Blind human or deterministic scoring
- Total time and billable usage
- Failure and recovery behavior
- Security and governance fit
It is a reason to lower confidence and run a controlled trial, not a reason to declare the vendor result false.
4. Build a ten-task test set that can reveal a bad fit
A useful test set contains tasks you would actually pay to complete. Keep it small enough to score deeply and varied enough that one model cannot win through a narrow trick.
| Task group | Example | Hidden failure to include |
|---|---|---|
| Routine x 2 | Small bug diagnosis, focused explanation | Simple work where orchestration overhead may not pay off |
| Repository x 2 | Cross-file regression, migration review | Misleading naming or an incomplete test suite |
| Research x 2 | Compare sources, reproduce a method plan | Conflicting evidence or a missing primary source |
| Long context x 2 | Contract analysis, design review | Relevant clause buried among plausible distractors |
| Adversarial x 2 | Underspecified request, impossible demand | The correct answer should ask, refuse, or state uncertainty |
Do not add a task after seeing which model failed it. Keep the test set frozen for the comparison, then create a new version for the next evaluation cycle.
5. Freeze the run contract
- Same task text and attachments for every model
- Same tool access, network access, and repository snapshot
- Same maximum wall time and output constraints
- Same retry count and retryable error policy
- Same system or developer instructions where supported
- At least three runs for tasks where nondeterminism could change the conclusion
- Raw response, status, duration, and usage retained for every attempt
If a field is accepted but ignored by one provider, do not pretend the run conditions are identical. Record the difference in the evaluation notes and focus on the user-visible contract.
6. Score the answer without showing the model name
Rename outputs A, B, and C before review. A simple 0 to 4 rubric is easier to apply consistently than a vague “felt better” judgment.
| Dimension | 0 | 2 | 4 |
|---|---|---|---|
| Correctness | Unsafe or fundamentally wrong | Partly right with material errors | Correct under the stated test oracle |
| Evidence | Unsupported or invented | Some traceable support | Claims map cleanly to sources or code |
| Completeness | Misses the main task | Covers core path, misses edge cases | Covers required behavior and key failure modes |
| Uncertainty | Confidently fabricates | Some caveats, not well calibrated | States limits and asks only necessary questions |
| Actionability | Cannot be used | Needs substantial interpretation | Clear next actions with verifiable outcomes |
Define the oracle before scoring. For code, it may be tests plus expert review. For research, it may be a source checklist. For a decision memo, it may be required questions and traceable evidence.
7. Add the dimensions a benchmark chart leaves out
Latency
Record median and 90th percentile wall time. A quality win can still fail an interactive service target.
Cost
Capture full usage, including orchestration fields. Report cost per successful task, not only cost per request.
Reliability
Count timeouts, tool failures, malformed outputs, and tasks requiring manual rescue.
Recoverability
Measure whether a failed run can resume safely or forces the whole task to repeat.
Observability
Ask whether logs and response metadata let an operator understand what happened without exposing sensitive content.
Governance
Check agent opt-outs, data handling, regional availability, audit needs, and whether a fixed pool is acceptable.
8. Make a narrow decision, not a universal ranking
A useful conclusion has the form “Use model X for task class Y under constraint Z.” It does not need one global winner.
Adopt Fugu when it meets the quality bar and its latency or control advantages matter.
Escalate to Ultra only where repeated blind scoring shows a material quality gain worth the extra time and usage.
Use Cyber only for authorized defensive security tasks after access approval, with scope and human review.
Keep your current model where orchestration adds no measurable outcome.
Keep self-managed orchestration when route visibility and deterministic governance are mandatory.
Re-run the test set after material model updates or workload changes. Sakana AI says it aims to update orchestration after new frontier model releases, so a decision can expire even if your application code does not change.
Primary sources
- Sakana AI Models: current score table, supported models, and behavior
- Sakana Fugu Technical Report: architecture, training, and reported evaluation methodology
- Sakana AI Fugu product page: benchmark notes and qualitative examples
Scores and product behavior are time sensitive. The evaluation process is designed to remain useful when the exact leaderboard changes.