85%with table and column names only
100%with written notes about the data
(+15 points)
7 → 15hard
questions right, of 15, once the notes are added
10 of 10unanswerable
questions refused, with notes
Accuracy by what the model was given
The notes do the work. Worked examples and a second try add nothing measurable on top: a difference of one question is within run-to-run noise.
The same ladder on other models
The ladder was first measured on DeepSeek, then again on gpt-5.4-mini. The jump comes from the notes on both.
| What the model was given | gpt-5.4-mini | deepseek-v4-pro |
|---|---|---|
| Table and column names only | 85% | 83% |
| + written notes about the data | 100% | 98% |
| + worked examples | 98% | 97% |
| + second attempt if the SQL fails | 100% | 98% |
Right answers by difficulty
Hard questions depend on a business rule or a trap in the data. Unanswerable questions should be refused.
| Run | Easy | Medium | Hard | Unanswerable | All |
|---|---|---|---|---|---|
| Table and column names only | 15 of 15 | 20 of 20 | 7 of 15 | 9 of 10 | 85% |
| + written notes about the data | 15 of 15 | 20 of 20 | 15 of 15 | 10 of 10 | 100% |
| + worked examples | 15 of 15 | 19 of 20 | 15 of 15 | 10 of 10 | 98% |
| + second attempt if the SQL fails | 15 of 15 | 20 of 20 | 15 of 15 | 10 of 10 | 100% |
| Everything, on deepseek-v4-flash | 15 of 15 | 17 of 20 | 15 of 15 | 10 of 10 | 95% |
| Everything, on deepseek-v4-pro | 15 of 15 | 19 of 20 | 15 of 15 | 10 of 10 | 98% |