Sample results
Same accuracy and the same review decisions on all 88 lines, at 4.5× lower cost per order.
43% less time per order on average
Both pipelines on 20 synthetic orders (88 lines), scored against a hand-reviewed answer key. Thresholds: product confidence 0.85, quantity clarity 0.80 (defaults).
Quality — no difference
Right productidentical
- Claude + Jev
- 87 of 8899%
- Claude only
- 87 of 8899%
Lines auto-approvedidentical
- Claude + Jev
- 66 of 8875%
- Claude only
- 66 of 8875%
Wrong among auto-approvedidentical
- Claude + Jev
- 0 of 660%
- Claude only
- 0 of 660%
With 66 lines, the true rate could still be up to ~4.5% (rule of three, approximate 95% upper bound).
Cost · whole pipeline
4.5× lower cost per order
| Pipeline | Per order (mean) | Per 10,000 orders |
|---|---|---|
| Claude + Jev | $0.0101 | $101.40 |
| Claude only | $0.0456 | $456.38 |
Excluding the 15 lines with no catalog match: 4.4× cheaper.
73 lines where Jev selected a catalog product. Reading and Claude batch matching costs are allocated equally across each order’s lines; Jev matching uses recorded per-line costs. The per-10,000 figures scale the unrounded mean.
By order
Approved = auto-approved lines. Right = product matches the answer key.
| Order | Lines | Needs review (answer key) | Jev approved | Jev right | Claude approved | Claude right | Jev unneeded reviews | Claude unneeded reviews | Jev cost | Claude cost | Open order |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Hendricks Framingo01 | 5 | 0 | 5 | 5 | 5 | 5 | 0 | 0 | $0.0108 | $0.0459 | Open |
| Nair Interiorso02 | 5 | 1 | 4 | 5 | 4 | 5 | 0 | 0 | $0.0113 | $0.0493 | Open |
| Silva Concrete & Formingo03 | 4 | 0 | 4 | 4 | 4 | 4 | 0 | 0 | $0.0100 | $0.0450 | Open |
| Lakeview Roofing Co.o04 | 4 | 0 | 4 | 4 | 4 | 4 | 0 | 0 | $0.0100 | $0.0451 | Open |
| Brandt Decks & Fenceso05 | 5 | 0 | 5 | 5 | 5 | 5 | 0 | 0 | $0.0107 | $0.0458 | Open |
| Ortega Drywallo06 | 5 | 0 | 5 | 5 | 5 | 5 | 0 | 0 | $0.0106 | $0.0457 | Open |
| Birchline Builderso07 | 5 | 0 | 4 | 5 | 4 | 5 | 1 | 1 | $0.0104 | $0.0489 | Open |
| Tse Renovationso08 | 5 | 0 | 5 | 5 | 5 | 5 | 0 | 0 | $0.0104 | $0.0455 | Open |
| Warmwall Insulationo09 | 6 | 0 | 6 | 6 | 6 | 6 | 0 | 0 | $0.0110 | $0.0460 | Open |
| Maki Post & Beamo10 | 6 | 0 | 6 | 6 | 6 | 6 | 0 | 0 | $0.0113 | $0.0466 | Open |
| Doyle Home Repairo11 | 3 | 2 | 1 | 3 | 1 | 3 | 0 | 0 | $0.0089 | $0.0436 | Open |
| Fischer Contractingo12 | 4 | 4 | 0 | 4 | 0 | 3 | 0 | 0 | $0.0093 | $0.0439 | Open |
| Parkside Renovationso13 | 4 | 1 | 3 | 4 | 3 | 4 | 0 | 0 | $0.0095 | $0.0444 | Open |
| Ruiz Handyman Serviceso14 | 4 | 4 | 0 | 3 | 0 | 4 | 0 | 0 | $0.0095 | $0.0441 | Open |
| Ridgeline Homeso15 | 3 | 0 | 3 | 3 | 3 | 3 | 0 | 0 | $0.0094 | $0.0443 | Open |
| North Shore Interiorso16 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | $0.0078 | $0.0423 | Open |
| Maple Grove Outdoor Livingo17 | 5 | 0 | 5 | 5 | 5 | 5 | 0 | 0 | $0.0113 | $0.0465 | Open |
| DeLuca & Sons Roofingo18 | 6 | 1 | 5 | 6 | 5 | 6 | 0 | 0 | $0.0110 | $0.0464 | Open |
| Pelletier Developmento19 | 4 | 4 | 0 | 4 | 0 | 4 | 0 | 0 | $0.0096 | $0.0470 | Open |
| Summit Custom Buildso20 | 4 | 3 | 1 | 4 | 1 | 4 | 0 | 0 | $0.0099 | $0.0465 | Open |
| Totals / mean cost | 88 | 21 | 66 | 87 | 66 | 87 | 1 | 1 | $0.0101 | $0.0456 |
Unneeded reviews: 1 of 88 lines flagged unnecessarily by Claude + Jev; 1 of 88 by Claude only.
Is the confidence trustworthy?
Share of lines with the right product, grouped by each pipeline's own confidence.
| Confidence | Lines | Right |
|---|---|---|
| >= 0.9 | 75 | 100% |
| 0.7 - 0.9 | 10 | 100% |
| < 0.7 | 3 | 67% |
| Confidence | Lines | Right |
|---|---|---|
| high | 72 | 100% |
| medium | 3 | 100% |
| low | 13 | 92% |
The right product was in Jev's 20-product shortlist for 100% of lines.
Lines either pipeline got wrong
How to read this
- Small sample: 20 orders, 88 lines. Read these as signals, not benchmarks.
- Parts of the pipeline, including Jev's quantity check, were designed after seeing these same orders. No held-out set has been run.
- Costs use recorded token counts at list prices, rather than billed dollars. The matched-lines estimate allocates shared reading and Claude batch matching cost equally across each order's lines; Jev matching uses recorded per-line cost.
- Results vary between API runs. Re-scoring saved outputs does not establish repeatability of new API runs.
- Claude only approves its own high ratings; sliders change Jev's gates. Both apply the same missing and large quantity rules.
- Excluding the 15 no-match lines that skipped Jev’s second call, the allocated whole-pipeline cost saving is 4.4×.
20 orders, 88 lines · synthetic data · answer key reviewed by hand: Mike Costanzo (2026-09-29) · thresholds: product 0.85, quantity 0.80; Claude: high ratings · models: claude-sonnet-5-5, jev-latest (resolved version not recorded) · pricing: lib/pricing.ts, list prices · date run: Original API run date not recorded. Full report: results/eval-summary.md in the repository.