An eval framework for an order-operations assistant
I own the evaluation framework for an internal assistant that answers natural-language questions about orders — payment method, fulfillment status, failure cause, SAP transmission contents — instead of support and QE manually querying four backend services. Built in Python with pytest, asyncio, and pydantic, it scores six dimensions: tool selection, factual accuracy, groundedness, completeness, hallucination, and refusal correctness.
Five of the six are deterministic rather than judge-based. Tool selection is precision, recall, and F1 against the expected tools in the logged call trace. Accuracy runs through typed normalizers — currency, storefront-local dates, order numbers, addresses, enums — then exact compare, which removed a large class of false failures caused by formatting rather than incorrectness. Hallucination is entity and numeric containment against the retrieved service payloads. Only the free-text explanation needs an LLM judge, and that judge is calibrated against the deterministic scorers on items with known ground truth, reported alongside its own agreement rate, and never allowed to gate a build alone.