How often is it right?

60 test questions, each with a checked reference query. A generated answer counts as right only if it returns the same result. Each run adds one thing to what the model is given.

Measured on gpt-5.4-mini, the model this site answers with, one run per row.

85%with table and column names only
100%with written notes about the data (+15 points)
7 → 15hard questions right, of 15, once the notes are added
10 of 10unanswerable questions refused, with notes

Accuracy by what the model was given

The notes do the work. Worked examples and a second try add nothing measurable on top: a difference of one question is within run-to-run noise.

  1. Table and column names only 85%
  2. + written notes about the data 100%
  3. + worked examples 98%
  4. + second attempt if the SQL fails 100%
  5. Everything, on deepseek-v4-flash 95%
  6. Everything, on deepseek-v4-pro 98%

The same ladder on other models

The ladder was first measured on DeepSeek, then again on gpt-5.4-mini. The jump comes from the notes on both.

What the model was givengpt-5.4-minideepseek-v4-pro
Table and column names only 85%83%
+ written notes about the data 100%98%
+ worked examples 98%97%
+ second attempt if the SQL fails 100%98%

Right answers by difficulty

Hard questions depend on a business rule or a trap in the data. Unanswerable questions should be refused.

RunEasyMediumHardUnanswerableAll
Table and column names only 15 of 1520 of 207 of 159 of 10 85%
+ written notes about the data 15 of 1520 of 2015 of 1510 of 10 100%
+ worked examples 15 of 1519 of 2015 of 1510 of 10 98%
+ second attempt if the SQL fails 15 of 1520 of 2015 of 1510 of 10 100%
Everything, on deepseek-v4-flash 15 of 1517 of 2015 of 1510 of 10 95%
Everything, on deepseek-v4-pro 15 of 1519 of 2015 of 1510 of 10 98%

What went wrong

For each review score, how many reviews include a written comment message? Return the score and the count.medium

Why it failed: wrong aggregation: counted distinct review_id because the notes say it repeats; the question is arguably ambiguous

Generated

SELECT review_score, COUNT(DISTINCT review_id) AS review_count
FROM order_reviews
WHERE review_comment_message IS NOT NULL
GROUP BY review_score
ORDER BY review_score

Reference

SELECT review_score, count(*) FROM order_reviews WHERE review_comment_message IS NOT NULL GROUP BY review_score