Eval numbers
Each workspace below opted to publish its latest eval run: a fixed set of questions with reference answers, asked to the live assistant. Refusals and citations are checked deterministically; a judge model grades correctness with a pinned rubric. No number here is typed in by hand.
Ali Shoes
Per-question results →gemini-3.5-flash-lite judged by gemini-3.5-flash-lite · rubric correctness-v1 · 40 cases · 14 Oct 2026
Answer accuracy
87%
26 correct · 3 partial · 1 incorrect of 30 answerable
Cited the right source
100%
answerable questions with an expected source
Correct refusals
100%
10 unanswerable questions
Over-refusal
3%
answerable questions the assistant refused
Avg latency
6535 ms
question to full answer
Run cost
$0.0277
40 cases · list prices, free tier pays $0
How the numbers are produced
- Answer accuracy: share of answerable questions the judge marked “correct” (every key fact present, nothing contradicting). “Partial” answers do not count.
- Cited the right source: share of answered answerable questions whose citations include the expected document.
- Correct refusals: share of unanswerable questions where the assistant declined instead of guessing. Over-refusal is the mirror on answerable questions.
- The judge is a different model family from the one under test when keys for both exist, to limit self-preference. Unparseable judge output counts as incorrect, never as missing.