How often is it right?

60 test questions, each with a checked reference query. A generated answer counts as right only if it returns the same result. Each run adds one thing to what the model is given.

Measured on gpt-5.4-mini, the model this site answers with, one run per row.

85%with table and column names only
100%with written notes about the data (+15 points)
7 → 15hard questions right, of 15, once the notes are added
10 of 10unanswerable questions refused, with notes

Accuracy by what the model was given

The notes do the work. Worked examples and a second try add nothing measurable on top: a difference of one question is within run-to-run noise.

  1. Table and column names only 85%
  2. + written notes about the data 100%
  3. + worked examples 98%
  4. + second attempt if the SQL fails 100%
  5. Everything, on deepseek-v4-flash 95%
  6. Everything, on deepseek-v4-pro 98%

The same ladder on other models

The ladder was first measured on DeepSeek, then again on gpt-5.4-mini. The jump comes from the notes on both.

What the model was givengpt-5.4-minideepseek-v4-pro
Table and column names only 85%83%
+ written notes about the data 100%98%
+ worked examples 98%97%
+ second attempt if the SQL fails 100%98%

Right answers by difficulty

Hard questions depend on a business rule or a trap in the data. Unanswerable questions should be refused.

RunEasyMediumHardUnanswerableAll
Table and column names only 15 of 1520 of 207 of 159 of 10 85%
+ written notes about the data 15 of 1520 of 2015 of 1510 of 10 100%
+ worked examples 15 of 1519 of 2015 of 1510 of 10 98%
+ second attempt if the SQL fails 15 of 1520 of 2015 of 1510 of 10 100%
Everything, on deepseek-v4-flash 15 of 1517 of 2015 of 1510 of 10 95%
Everything, on deepseek-v4-pro 15 of 1519 of 2015 of 1510 of 10 98%

What went wrong

Which category has the most items sold? Return the category and the number of items.medium

Why it failed: wrong filter: added a delivered-only filter the question did not ask for (copied from the revenue example)

Generated

SELECT t.product_category_name_english AS category, COUNT(*) AS items_sold
FROM order_items i
JOIN orders o ON i.order_id = o.order_id
JOIN products p ON i.product_id = p.product_id
JOIN product_category_name_translation t ON p.product_category_name = t.product_category_name
WHERE o.order_status = 'delivered'
GROUP BY t.product_category_name_english
ORDER BY items_sold DESC
LIMIT 1

Reference

SELECT t.product_category_name_english, count(*) AS items FROM order_items i JOIN products p ON i.product_id = p.product_id JOIN product_category_name_translation t ON p.product_category_name = t.product_category_name GROUP BY 1 ORDER BY items DESC LIMIT 1