Top pickBuilding with AI
Build an eval set
Creates test cases and grading criteria for an LLM feature, so prompt changes are measured, not guessed.
Use it when
Before changing a prompt or model in production.
What you fill in
- feature What the feature does
- failures Known failures and user complaints
The prompt
Design an eval set for: feature Known failures: failures 1. Success criteria: 4 to 6 measurable properties of a good output. 2. 30 test cases: 15 typical, 10 edge cases, 5 adversarial. For each: the input, what a good output must contain, and what it must not. 3. A grading rubric per criterion, deterministic checks first (format, length, required fields), then a model-graded check with the exact grader prompt. 4. How to run it on every prompt change and the pass threshold to ship. Before you answer, check your draft against every instruction above and fix anything that misses.
Why it works
- Defines success before testing.
- Mixes typical, edge and adversarial cases.
- Prefers deterministic checks and specifies the grader prompt.