Why We Replaced 'Extract' With 'Transcribe' (And Got 7x More Data)

The setup

FieldIntel reads handwritten field books – the bound notebooks surveyors carry into the field – by sending each page to a vision-language model (VLM) running locally under Ollama. By default we ship with llama3.2-vision, which is small enough to run on a single consumer GPU and capable enough to read most legible handwriting.

VLMs are language models with an image input head. That matters, because it means they don’t behave like an OCR engine even when you want them to. They want to help. They want to summarize, organize, smooth over what’s on the page, and produce something polished. For a handwritten field book – where the value is in the messy specifics, not the gist – helpful is exactly wrong.

The original prompt

Our v1.0 prompt was the kind of thing you’d write if you’d just read a prompt-engineering blog post:

You are an expert land surveyor reading a scanned field book page.
Extract and label the following if present:
- Job/Project Number
- Date(s)
- Point numbers with their coordinates (N, E, Z)
- Descriptions / feature codes
- Handwritten notes or annotations
- Geometric sketches (describe in detail)

Format your response with clear headings for each category.

The intent was reasonable: tell the model what to look for, ask it to organize the output so a downstream parser can find structured data under known headings. The “expert land surveyor” role assignment was meant to anchor the model in domain context.

In retrospect, both halves of that prompt are wrong.

What triggered the rebuild

We were reviewing an ingested archive – about 3,000 pages from a pilot customer’s historical archive – against the source PDFs and noticed a consistent pattern of confident-sounding nonsense in the extracted text. A few examples:

• A page-header date written as 2-11-25 was rendered as 1985.
• The phrase TOAD EARTH appeared ten times in the extractions from one page; it does not appear anywhere on that page.
• Coordinate sequences fabricated outright – point numbers and (N, E) pairs that simply don’t exist on the source page.
• Sketches described in elaborate prose that bore no resemblance to the actual sketch.

None of these were OCR errors. They were the model generating “what a survey field book page should look like,” anchored loosely to the visual content. The page was a Rorschach test and the model was telling us what it saw in the inkblot.

The experiment

We built a small harness in experiments/vlm_quality/ that does the same thing the production ingest does – render PDF page at the same DPI, send to the same Ollama endpoint, score the output by an entity-extraction regex – but lets us run multiple prompts over the same test set and compare outputs side by side.

Twenty regions sampled at random across the pilot archive. Same model (llama3.2-vision). Same render. Same entity scorer. Two prompts:

Coverage – the count of regions where some signal was extracted – was effectively a wash. Entity density on the regions that did produce signal jumped from 1.4 per region to 10.4 per region: a 7.4x lift on covered regions, a 6.7x lift on the test set as a whole.

Wall-clock latency was within about 10 percent across the two prompts on the same machine. The replacement is not faster, not running on better hardware, not using a more capable model. It is the same model doing a narrower job.

The replacement prompt

Transcribe the text in this scanned field book region VERBATIM.
Preserve original formatting (line breaks, separators like | and :).
Do NOT summarize. Do NOT paraphrase. Do NOT add commentary or
interpretation.

If the region contains:
- Point records (e.g. "Pt 100036 | Elev 660.572 | (39.31, -85.95)"),
  preserve the EXACT format.
- Job numbers, dates, elevations, coordinates: write them EXACTLY
  as they appear, including punctuation.
- Feature codes (IR, REB, MON, EOP, etc.): write them in CAPS
  exactly as they appear.
- Sketches: write a brief description in [brackets] but transcribe
  any text labels VERBATIM.
- Illegible text: write [illegible] -- do not guess.

Begin transcription now (no preamble, no headers, just the
transcribed content):

Note what is not in this prompt. There is no role assignment (“you are an expert…”). There is no request to organize, label, or categorize. The example format is concrete (Pt 100036 | Elev 660.572 | (39.31, -85.95)) rather than abstract (“point records”). The instruction “do not guess” is explicit, with [illegible] as the escape valve.

Why this works

A handwritten field book has dense local context – the surveyor’s handwriting, the gridded paper, the column headers Pt | Elev | Description – and very little global context for the model to fall back on. The original prompt asked the model to look at this low-context, high-noise content and produce something polished and organized. With nothing to organize into, the model produced its best guess at “what an organized survey-page summary should look like.” That is why we got the hallucinations. The model was generating, not reading.

The replacement prompt narrows the job to a single mechanical task: transcribe what you see. There is no room for the model to be helpful, because helpfulness is not part of the task. The structuring step moves to a downstream regex pass over the transcription, which is dumb and fast and either matches or doesn’t.

This is a different shape of pipeline than “give the LLM the whole job and trust it.” For handwritten field notes specifically, the LLM is not the smartest component in the pipeline – it is the eyes. The brain is regex.

What shipped

FieldIntel Server v1.2.2 ships with the transcription prompt as the default vlm_prompt in the bundled fieldintel.yaml. No model swap. No infrastructure change. No schema change. Existing customers can update by pulling the new template config or by editing vlm_prompt: in their existing config to match.

Already-ingested archives are not retroactively reprocessed. Field notes ingested under the old prompt have whatever entity coverage they had when they were ingested. Customers who want to recover that data can re-run ingest on the affected files; making it a one-button operation is on the v1.3 list.

What we didn’t test

This is a one-model, one-document-type result. We have not tested:

• Whether the same effect holds on qwen2.5vl or minicpm-v. Qwen errored out repeatedly during the test run – we’ll come back to a model comparison in a later post.
• Whether the effect holds on engineering plans, plats, or other non-field-note document types. Those use a different prompt path that hasn’t been audited yet.
• Whether more aggressive prompt restructuring (chain-of-thought, few-shot examples, hard structured-output constraints) would lift the number further.

The takeaway is narrower than “transcription always beats extraction.” It is that for handwritten field notes, on the model FieldIntel ships by default, telling the VLM to transcribe and parsing the output yourself produces 6.7x more usable data than asking the VLM to extract and organize in one step.

The general lesson, if there is one: when the model is not smart enough to do two things well, ask it to do one thing well and do the second thing yourself.