QASubsection
AI testing
When the same input stops producing the same output, an equality assertion becomes meaningless and a single run stops being evidence. Three topics: what that changes for the person writing the cases, how to judge one answer without spending money to replace a fact with an opinion, and how to tell a real regression from sampling noise. Running an LLM feature in production is LLMOps, in Development — these three own testing it.
What is inside
Every page in this group, with what each one covers.
- Testing LLM ProductsBuild the deterministic layer first — schema, tools, citations, safety, budget — then who owns what, and where each check runs.Deterministic firstPipeline
- AI EvaluationThe grader ladder from exact match to a human, judge calibration, and how to read a rate without fooling yourself.Eval setsJudges
- Prompt RegressionThe per-case baseline, the four things that must be pinned, the noise floor, and the protected set that blocks a release.BaselinePinning
Where to start
- Testing an LLM featureTesting LLM products. Most of what matters is still exactly assertable, and that layer is where to spend the first week.
- Building an eval setAI evaluation — the grader ladder first, because most sets are graded far higher up it than they need to be.
- Changing a promptPrompt regression. Compare case by case against a recorded baseline — the aggregate hides the cases that broke.
Elsewhere in the section
- QAQuality assurance is the practice of deciding what to check, running those checks often enough to be worth having, and knowing what changes when the thing under test stops giving the same answer twice. This section covers 9 topics in three groups: the types of testing everyone half-remembers, automation with Playwright, Cypress and Selenium, and testing products built on language models — where an assertion becomes a score rather than an equality.
- AI & LLMBuilding features on a component that gives a different answer to the same question twice. Six topics in the order they become relevant: what the model is actually doing, how to ask it properly, how to give it your own data, when to change the model itself, how to run the result in production, and what changes when it can call tools. Testing these features is a discipline of its own and lives in QA.
- Types of TestingThree tests that people put on one list and that are not comparable. Smoke is a gate: it decides whether a build is worth anyone else’s time. Regression is memory: it remembers what already broke once. End-to-end is a level: it is the only place the assembled system can be proved to work. The first two are reasons to run tests and can be written at any level, which is the distinction most interview answers miss.
Found this useful?
Share it with someone who is working on the same problem.