Summary
SDET / AI Quality Engineer with 7+ years in test automation, focused on evaluation infrastructure for LLM and agentic systems. I build objective, repeatable evaluation harnesses in Python that score tool selection, groundedness, accuracy, completeness, and hallucination — scoring deterministically wherever ground truth is computable and reserving LLM-as-a-judge for what genuinely needs judgment. I own benchmark dataset design and maintenance, CI/CD quality gates, and regression detection across prompt and model changes, on a backend automation foundation of Java, Python, RestAssured, Playwright, and Selenium.
Experience
AI Quality Engineer / SDET · Apple
- Own the evaluation framework for an internal order-operations assistant used by support operations and QE to answer natural-language questions about orders — payment method, fulfillment status, failure cause, SAP transmission contents — instead of manually querying four backend services.
- Built the framework in Python (pytest, asyncio, pydantic), scoring six dimensions: tool selection, factual accuracy, groundedness, completeness, hallucination, and refusal correctness.
- Made five of six dimensions deterministic rather than judge-based: tool selection as precision/recall/F1 against expected tools from the logged call trace; accuracy via typed normalizers then exact compare; hallucination as entity and numeric containment against retrieved service payloads. Only free-text explanation uses an LLM judge.
- Designed a 600-item benchmark over 200 seed orders across six storefronts, order types, and payment methods, with golden answers derived from known order parameters rather than hand-labeled — making ground truth reproducible instead of an annotation cost.
- Solved golden-answer staleness by classifying every field immutable vs volatile and re-deriving volatile expected values before each run; changes to immutable fields surface as data defects, not eval drift.
- Handle non-determinism with n=5 sampling, pass-rate scoring instead of pass/fail, and a tracked consistency metric, isolating unstable items so eval flakiness is never reported as a model regression.
- Calibrated the LLM judge against the deterministic scorers on items with known ground truth, reporting judge agreement alongside its scores and never gating builds on the judge alone.
- Integrated evaluation into CI/CD: stratified 120-item runs on every PR touching prompts, tool definitions, or model version, and full 600-item nightly suites, with per-dimension deltas posted to the pull request on pass and fail alike.
- Implemented tiered quality gates — hard failure on hallucination rate, tool-selection F1, and accuracy; warnings on completeness and consistency — after an all-or-nothing gate trained the team to re-run until green.
- Caught a model-version regression in which overall accuracy improved while tool-selection F1 fell on multi-tool questions, causing stale data to be reported confidently; resolved by clarifying tool staleness semantics in the tool definitions.
- Develop and maintain MCP server integrations for agent tooling, testing tool registration, call routing, malformed-argument handling, error states, and timeout propagation as a suite independent of agent reasoning.
- Automate backend validation across 30+ service clients over gRPC and REST with OpenAPI/YAML-driven specs, covering contract, negative, edge-case, and error-code paths including XML/JSON verification and authentication flows.
Software Development Engineer In Test · Elevance Health
- Owned test documentation and strategy — test plans, scripts, cases from acceptance criteria, scenarios, and automation — across the full SDLC.
- Drove sanity, functional, and system testing on AWS, delegating execution and validation across three team members to meet delivery deadlines.
- Implemented and managed CI/CD pipelines with automated testing, reducing deployment time by 40%.
- Automated microservice delivery workflows with RestAssured and Java, plus database validation for REST/GraphQL architectures via Postman, load testing in JMeter, and security regressions in Jenkins.
- Partnered with engineering and DevOps to integrate automated validation into pipelines while improving logging, observability, and defect analysis across platform services.
Quality Assurance Engineer · Discover Financial Services
- Automated UI regression and smoke suites with Selenium WebDriver, cutting testing time by 40%.
- Built a Cucumber-and-Java framework for back-end and database testing, using RestAssured for RESTful APIs over HTTP and HTTPS.
- Implemented a validation utilities layer for config-driven API tests: request/response comparison, schema-level assertions, and expected vs actual checks.
- Used JUnit for unit and integration testing, JDBC for database interactions, and Gradle for build automation; maintained the requirements traceability matrix in JIRA.
Selected impact
600Benchmark items over 200 seed orders
5/6Eval dimensions scored deterministically
30+Service clients automated over gRPC and REST
-40%Deployment time via CI/CD automation
Technical stack
- Evaluation
- Benchmark dataset design and versioning, derived golden answers, stratified sampling, deterministic scorers vs LLM-as-a-judge, judge calibration, multi-sample pass-rate scoring, consistency tracking, per-dimension regression reporting
- AI & agentic
- MCP servers, multi-turn tool-calling validation, structured outputs, Anthropic Messages API, cost and latency tracking, adversarial and prompt-injection testing
- Languages
- Python, Java, JavaScript, TypeScript, SQL, YAML, Gherkin
- Automation
- pytest, Playwright, Selenium, RestAssured, Cucumber, TestNG, JUnit, Karate, Postman, JMeter, Allure, TestRail
- Interfaces & data
- REST, gRPC, GraphQL, OpenAPI/Swagger, MySQL, Oracle, MongoDB, Snowflake (Cortex Analyst), SAP IDoc validation
- Cloud & CI/CD
- AWS, Azure DevOps, Docker, Kubernetes, Jenkins, GitHub Actions, Maven, Gradle, Allure and Slack automated reporting
- Practices
- Eval-driven development, risk-based testing, stage quality gates, release ownership, go/no-go decisions, Agile/Scrum, TDD, BDD, Page Object Model
Elsewhere
temirtilla.com linkedin.com/in/temirn youtube.com/@temirtilla temir.nurlanbek@gmail.com
Last updated August 2026 · The live version is always at temirtilla.com/resume