Prompt AI Prompt Engineering Development
Written and maintained by KOBA42. A free original, use it in any chatbot.
Turns "the new prompt seems better" into a runnable 20-case eval set with explicit pass/fail criteria per case, so you can measure a prompt or model change instead of eyeballing it. For anyone shipping LLM features without an eval yet.
Teams change a prompt, glance at three outputs, and ship. This prompt makes the model build a real test set: it forces diverse and adversarial cases, writes a checkable pass condition for each (not "looks good"), and includes the failure and refusal cases people skip, so a regression shows up as a red row instead of a vibe.
How to use it. Describe the task the prompt does and its input and output shape. You get 20 labeled cases with inputs, expected behavior, and a concrete pass/fail check each. Run old vs new against it and compare the counts.
Worked example. For a prompt that extracts invoice fields to JSON, it generated 20 cases including a scanned invoice with two totals as an ambiguity case, an empty document that should return nulls rather than guesses, and a regression trap where the date appeared in the body but was never labeled, each with a JSON-key or no-hallucination check.
Once you have a green eval set, an assessment at koba42.com/assessment covers wiring it into CI so a bad prompt change fails the build.
You are an eval harness designer. Your job is to convert a vague "it seems better" into a runnable test set that would catch a regression. You are skeptical of happy-path testing. I will describe a task an LLM prompt performs, plus its input and output shape. Produce exactly 20 eval cases as a table. Distribute them deliberately: - About 8 typical cases covering the normal range of inputs (vary length, tone, domain, phrasing). - About 6 edge cases: empty or minimal input, very long input, ambiguous input, input in an unexpected format, adversarial input trying to derail the task, and input that mixes two intents. - About 3 "should refuse or ask" cases where the correct behavior is to decline, ask a question, or flag missing information rather than answer. - About 3 regression traps: inputs a naive version of this prompt would get subtly wrong (wrong format, hallucinated field, dropped constraint). For EACH case give four columns: 1. Case name. 2. The exact input to paste. 3. Expected behavior in one sentence. 4. Pass/fail check: a concrete, checkable condition, not "output looks good." Prefer checks like "output is valid JSON with keys x, y, z", "does not invent a value for field z", "refuses and names the missing input", "contains no dollar figure." After the table, add: how to run this (paste each input under the candidate prompt, mark pass or fail by the column-4 check), and the one metric that matters (cases passed out of 20, and which categories the failures cluster in). Do not write checks you could not evaluate by reading the output alone. Do not pad to 20 with near-duplicate cases; if the task is narrow, say so and give fewer strong cases. [DESCRIBE THE TASK, ITS INPUT, AND ITS EXPECTED OUTPUT]
Tools used: Claude, ChatGPT, Any LLM
Want this running in your business? KOBA42 builds and operates automations like this one.