Gemini 3 Flash vs Gemini 2.5 Flash: A Measured Regression

We moved one production step — script extraction — from Gemini 2.5 Flash to Gemini 3 Flash, and the pass rate fell from 97.2% to 55.6%. Gemini 3.5 Flash landed at 63.9%. Same prompt, same schema, same nine fixtures, same day.

The newer models are not worse in general. They are worse at this task, and that distinction is the whole point: a model's headline benchmarks tell you almost nothing about how it will behave on the narrow, schema-bound job you're actually using it for.

What we measured

Script extraction is the step that turns a raw script into structured voice blocks — the units that later become narration audio. It's a preservation problem more than a reasoning one. The model has to segment the text without silently altering it.

The harness runs nine fixtures (narration, mixed dialogue, listicles, wildlife profiles, heist and factory scenes with implied characters) four times each — 36 trials per model. A metric only passes if it holds on every trial, because an extractor that works three times in four is an extractor that ships a broken video once in four.

There's no LLM judge anywhere in this. Every metric is an exact count:

• Break preservation — every <break /> tag in the source survives byte for byte.
• Audio tag preservation — every [sighs]-style tag survives verbatim.
• No orphan blocks — no block whose text is only a tag.
• No empty blocks — no block with empty text.
• Coverage — the concatenated blocks still cover the source.

Objective counts matter here. A judge model would have introduced exactly the kind of variance the test exists to detect.

The result

Nine fixtures, four trials each, control prompt, run on 2026-07-07:

Two fixtures went from perfect to zero. adventure-props scored 4/4 on the older model and 0/4 on both newer ones.

What actually broke

The failures weren't spread evenly across the metrics — they concentrated in one:

An orphan block is a voice block whose text is nothing but a tag — a lone <break /> with no words around it. It looks harmless in a JSON payload. Downstream it isn't: an orphan tag renders as a standalone clip, so you get a silent fragment sitting in the timeline where a sentence should be.

The newer models were also more prone to over-extraction — pulling one-off nouns out as though they were recurring entities. On adventure-props, that meant treating a passing gull and a teacup as tracked objects, on every trial.

So the regression isn't "the model got dumber." It's that the newer models segment more aggressively, and aggressive segmentation is precisely wrong for a task whose job is to preserve.

Why a newer model can score worse

Newer models are tuned against broad public benchmarks, and each Gemini generation targets a wider capability envelope than the last. Yours is a narrow, schema-constrained task with a preservation requirement — and nothing in a general benchmark suite rewards restraint of the kind this needs.

There's a second effect worth knowing, because it's counterintuitive: lowering the temperature didn't help. Running the control at temperature 0.2 made over-extraction worse, not better, because a lower temperature makes a consistent bias more consistent. If the model is reliably wrong, determinism just makes it reliably wrong every time. The fix was a prompt clause, and the clause and the temperature had to ship together.

What we do now

The rule we landed on is written into the harness itself: do not bump the model without re-running this suite. Not as a policy statement — as a comment on the constant that sets the model, where whoever changes it will read it.

The broader practice:

1. Keep a fixture set that represents the shapes your task actually sees, not just the easy case. Two of our nine fixtures caught the whole regression; the other seven looked fine on all three models.
2. Score with exact counts wherever the task allows it. Reserve LLM judges for genuinely subjective properties.
3. Require every trial to pass, not the average. Averages hide the intermittent failures that reach users.
4. Re-run on every model version bump, and date the result. Ours is dated because it will expire — Google lists current and legacy Gemini versions separately for exactly this reason, and revisions ship under names that don't always change.

That last point is the honest limitation here. This is one task, one prompt, one schema, measured on one day. It is not a claim about Gemini generally, and a later revision of either model could reverse it entirely. The transferable finding isn't which model won — it's that a version bump is a change that needs testing like any other, and that "newer" is not a synonym for "better on your task."

How this fits the pipeline

framesail runs a six-station pipeline where every station is a separate model call, and each one has a suite like this behind it. Extraction feeds voiceover, which feeds timing, which feeds the storyboard — so a 41-point drop at station one doesn't stay at station one. That's the argument for treating the pipeline as the unit of quality rather than any single model, and it's the same reason character references are locked rather than re-prompted.

To try the pipeline, start a project.

FAQ

Is Gemini 3 Flash worse than Gemini 2.5 Flash?

On our script-extraction task, measurably yes — 55.6% versus 97.2% across 36 trials. On general capability, no; the newer models are stronger on the broad benchmarks they're tuned against. The gap between those two statements is the reason to run your own evaluation rather than reading a leaderboard.

What is an orphan block?

A structured output block whose text contains only a tag and no words — a lone <break />, for example. It passes schema validation because the field isn't empty, then misrenders downstream as a standalone clip. It's the kind of failure that's invisible in the JSON and obvious in the finished video.

Why not use an LLM judge to score this?

Because the task has an objective answer. Break tags either survive byte-for-byte or they don't. Introducing a judge model would add variance to a measurement whose entire purpose is detecting variance.

Does a lower temperature fix over-extraction?

It didn't for us — it made it worse. Temperature controls variance, not bias. A model that consistently over-extracts will do so more consistently at a lower temperature. The fix was an explicit prompt constraint, shipped together with the temperature change.

How often should you re-run a model evaluation?

On every version bump, at minimum. Providers ship revisions under the same model name, so "we tested this" has a shelf life measured in weeks. Dating the result is what makes it honest.

How many fixtures do you need for this to be meaningful?

Fewer than you'd think, if they're chosen for variety of shape rather than volume. Nine fixtures at four trials caught a 41-point regression here — and two of the nine did most of the work. Coverage of failure modes beats sample size.