QASubsection
AI testing
When the same input stops producing the same output, an equality assertion becomes meaningless and a single run stops being evidence. Three topics: what that changes for the person writing the cases, how to judge one answer without spending money to replace a fact with an opinion, and how to tell a real regression from sampling noise. Running an LLM feature in production is LLMOps, in Development — these three own testing it.
What is inside
Every page in this group, with what each one covers.
- Testing LLM ProductsTwenty failures an LLM feature ships with and the check for each — fourteen of them exact — then properties, the 2026 OWASP list and five releases.Deterministic firstOWASP 2026
- AI EvaluationSixteen things to grade and the cheapest grader for each, error analysis first, judges reported with TPR and TNR, and four numbers that misled.Error analysisJudges
- Prompt RegressionOne prompt change reviewed end to end, negative flips, the six inputs to pin, three runs a case, the gate and the model upgrade.Negative flipsPinning
Where to start
- Testing an LLM featureTesting LLM products. Most of what matters is still exactly assertable, and that layer is where to spend the first week.
- Building an eval setAI evaluation — read a hundred real conversations before choosing a metric; most failure groups turn out to be checkable by code.
- Changing a promptPrompt regression. Compare case by case against a recorded baseline — the aggregate hides the cases that broke.
Elsewhere in the section
- QAQuality assurance is the practice of deciding what to check, running those checks often enough to be worth having, and knowing what changes when the thing under test stops giving the same answer twice. This section covers 9 topics in three groups: the types of testing everyone half-remembers, automation with Playwright, Cypress and Selenium, and testing products built on language models — where an assertion becomes a score rather than an equality.
- AI & LLMBuilding features on a component that gives a different answer to the same question twice. Six topics in the order they become relevant: what the model is actually doing, how to ask it properly, how to give it your own data, when to change the model itself, how to run the result in production, and what changes when it can call tools. Testing these features is a discipline of its own and lives in QA.
- Types of TestingThree tests that people put on one list and that are not comparable. Smoke is a gate: it decides whether a build is worth anyone else’s time. Regression is memory: it remembers what already broke once. End-to-end is the journey: it is the only place the assembled system can be proved to work, from start to finish. The first two are reasons to run tests and can be written at any level, which is the distinction most interview answers miss.
Found this useful?
Share it with someone who is working on the same problem.