The short version
If you can get structured data through tool use (function calling), don’t beg for it with “please answer in JSON”. The first has the model filling in blanks under a schema. The second is you betting it’s in a good mood today.
Here are the traps I hit again on almost every project.
Trap 1: Asking for JSON in prose
"Return only JSON, no explanation" — and back comes “Sure, here’s what you asked for:” up front, a json code fence in the middle, and “want me to adjust anything?” at the end. Explanations, pleasantries, Markdown fences, the whole set.
Two levels of fix:
- Best: define the output as a tool’s
input_schemaand let the model go through tool use. What comes back is structured arguments, with no wrapper text by construction. - Fallback: use native JSON mode. With Claude you can prefill a
{on the assistant turn to force it to start from the object; with OpenAI,response_format.
Stop writing regexes to dig {...} out of text. That’s a landmine you’re leaving for yourself.
Trap 2: Fields drift
The format being right doesn’t mean the content is stable. Run the same prompt 100 times and you’ll see:
- optional fields that come and go
countsometimes3, sometimes"3"- enums improvised: you asked for
"high" | "low", it hands you"medium-high"
None of this shows in a demo. It blows up at volume. The fix is to put the constraints in the schema, not in polite wording in the prompt: mark required fields required, list every enum value explicitly, declare numbers as type: number. A schema is a contract written for a machine to read. It works far better than “please try to keep the format correct”.
Trap 3: Truncated JSON
The output hits max_tokens, the JSON stops halfway, and JSON.parse throws.
- Leave token headroom. Better to waste some than to sit right on the edge
- When you need a long list back, have the model report a
totalfirst, then ask for it in batches - If you really want to render while it streams, use an incremental parser that tolerates incomplete input instead of waiting for the whole blob
Trap 4: Wanting reasoning and clean JSON at once
The most common self-inflicted wound: asking the model to “analyze this first, then give the conclusion in JSON”. Either it stuffs a wall of analysis into some field and pollutes your data, or it stops thinking to keep the format clean and the conclusion gets worse.
Split reasoning and structuring into two steps: let it reason freely in the first, and have the second do nothing but “turn the conclusion above into JSON”. One extra call’s worth of tokens, and you stop debugging dirty data.
One line
The goal of structured output isn’t “it looks like JSON”, it’s “data I can feed straight downstream with no cleanup pass”. Tool use covers the format, the schema constrains the content, and the rest is on you not being lazy and collapsing two steps into one.