Does saying 'return only JSON' really get you clean JSON?

The question

When you need data a program can read, does adding 'Return ONLY a JSON object, no explanation, no code fences' change how often the whole answer is valid JSON?

What we found

• Haiku: 7% to 100% (strict did better)
• Sonnet: 0% to 100% (strict did better)

"Clear" means the two 95% ranges below do not overlap. When they overlap we say there is no clear difference, even if the percentages differ. With about 30 answers per cell, small gaps are noise.

Model	How we asked	Passed	Rate	95% range
Haiku	Asked for JSON in one casual sentence	2 of 30	7%	2% to 21%
Haiku	Told: JSON only, no prose, no code fences	30 of 30	100%	89% to 100%
Haiku	plain, valid JSON after pulling out the first {...} block	30 of 30	100%	89% to 100%
Haiku	strict, valid JSON after pulling out the first {...} block	30 of 30	100%	89% to 100%
Sonnet	Asked for JSON in one casual sentence	0 of 30	0%	0% to 11%
Sonnet	Told: JSON only, no prose, no code fences	30 of 30	100%	89% to 100%
Sonnet	plain, valid JSON after pulling out the first {...} block	30 of 30	100%	89% to 100%
Sonnet	strict, valid JSON after pulling out the first {...} block	30 of 30	100%	89% to 100%

What counts as a pass: answer is valid JSON as written, with the right values.

What we saw reading the answers

• Without the instruction, almost every answer came wrapped in a code fence (a ```json block). The values inside were correct every time: all 120 answers were right once we pulled the JSON out. Only the wrapping broke a strict parser.
• With "return ONLY a JSON object, no explanation, no code fences", every answer from both models was plain JSON that parsed as written.
• What to do: ask for JSON only and say "no code fences". Still strip fences in your code, because other models and other days will not always obey.

The exact wording

Shown with the first question as the example (the long passage is cut short here).

Asked for JSON in one casual sentence

System prompt: You are a helpful assistant.

Extract the product name and the sentiment (positive or negative) from this review and give it to me as JSON with keys "product" and "sentiment".

Review: [passage of 12 words]

Told: JSON only, no prose, no code fences

System prompt: You are a helpful assistant.

Extract the product name and the sentiment (positive or negative) from this review.
Return ONLY a JSON object with keys "product" and "sentiment". No explanation, no markdown, no code fences.

Review: [passage of 12 words]

How we tested

• Models: Haiku (claude-haiku-5-5); Sonnet (claude-sonnet-5-5). Claude models only. We have not tested ChatGPT, Gemini, Grok or others, and results can differ.
• 10 different questions, each asked 3 times per way of asking, so the answer's randomness is included.
• Every answer was scored by a short program, not by a person's opinion. We then read a sample by hand.
• Each question was sent on its own through the Claude command-line tool, with tools, memory, settings and conversation history switched off, and no web access or examples. That tool still adds a short fixed note about the machine (operating system, working folder, the account's email domain), about 900 tokens in all. It is identical for both ways of asking, so comparisons stay fair, but absolute numbers may differ a little from a plain API call. We redacted the owner's company name from one published answer where a model repeated it.
• Run on 2026-10-26. Model versions change; treat results as a snapshot.

Limits

Ten short product reviews. Easy extraction; harder inputs may behave differently.

Raw data

Every question, answer and score is in the raw data file (filter by "exp": "json-only"). Re-run it yourself with the harness in the project's testlab folder, or paste the prompts above into your own account.