Why Document AI Highlights the Wrong Place on the Page

A reviewer clicks an assistant's citation for a delivery date. The date is transcribed correctly, but the highlighted rectangle covers a different paragraph. The extraction service analyzed a cropped, reduced image; the viewer drew its coordinates on the original scan. Nothing in the answer's fluent wording reveals the mismatch.

This constructed example describes an integration failure, not a reported customer incident. For teams building document assistants, the decision is practical: when is an exact highlight trustworthy enough to show, when is a page-only reference more honest, and when should processing stop for review? A location is evidence only if the application can trace it back to the correct source version.

Text, location and meaning are three different checks

Recognition asks whether the characters were recovered. Localization asks where those characters occur. Interpretation asks whether they support the answer. These checks can fail independently. A correct date highlighted in the wrong paragraph fails localization; a perfectly aligned date from an expired amendment may fail interpretation.

Do not collapse these outcomes into one “OCR confidence” value. A recognition score does not establish that your own crop adapter, page-number conversion or display transform is correct. Those components may operate after the extractor has finished. Changing the recognition model can leave the same mapping bug untouched.

Our broader multimodal document-intelligence guide treats the source page as an evidence layer. This article develops the narrower implementation problem: carrying a location across file versions and coordinate systems. The recommendations and examples below are ZharfAI's engineering analysis, not a benchmark of the cited products or a guarantee of their accuracy.

Give coordinates a frame before drawing them

The tuple 0.25, 0.50, 0.20, 0.10 is not self-describing. It could mean a relative rectangle, physical units, or four points interpreted incorrectly. A useful coordinate record names the document version, page, reference image or page frame, dimensions, unit, origin, axis directions and geometry type. The producer's API version belongs alongside it.

Amazon Textract's BoundingBox reference defines left, top, width and height as ratios of page dimensions, with the origin at the upper left. A ratio must be expanded against the dimensions of the page that the result describes—not whichever thumbnail happens to be available.

Azure Document Intelligence's response guide describes bounding regions with one-based page numbers and polygon points in the page's declared unit: generally pixels for images and inches for PDFs. Read that unit instead of copying another provider's conversion formula. Preserve a polygon as a polygon rather than treating its first four numbers as rectangle dimensions.

Make each provider adapter explicit. It should decode its own response into your chosen frame and preserve the original geometry for diagnosis. A shared field named “box” does not make two schemas interchangeable. Missing unit or page identity is a failed contract, not an invitation to guess from the apparent size of the numbers.

Normalize orientation once and record what changed

A phone image can carry orientation metadata as well as stored pixels. Different processing stages may interpret that metadata differently. Pillow's exif_transpose documentation describes applying a non-default EXIF orientation to the image and removing the orientation data. That is a concrete example of turning an orientation instruction into changed pixels.

Choose an ingestion convention and record its result. If you normalize first, send that derivative to recognition and identify it in the extraction record. Keep the authorized source and the mapping to it. Do not subsequently rotate the returned coordinates again merely because the original upload once had an orientation tag.

Treat this as a chain of named frames: original source, orientation-normalized image, cropped image, resized analysis image, then display surface. Store the actual dimensions after each operation. A right-angle rotation can exchange width and height; a deskew operation may add margins. The old dimensions are not a reliable description of the new image.

The extractor may also perform internal preprocessing. Do not infer its output frame from how you imagine that preprocessing works. Follow the documented response contract and test the specific processor and version. Your record must distinguish operations your application performed from transformations reported by a provider.

Keep a small transformation record with the evidence

For each analyzed derivative, retain a source-version identifier, a content digest for integrity checks, the physical page index, derivative dimensions, and an ordered list of transformations. Include crop bounds, scale factors, padding, orientation decisions and the relevant library versions. A file name alone is insufficient: a replacement upload can reuse it.

Use a named internal page-number convention and convert at adapters. Printed page labels, the physical page position in a PDF, and an array index are different things. A cover sheet labeled with a Roman numeral should not shift every citation by one. When a job splits a document into page images, carry the original page association into each job.

Keep immutable extraction versions. If a rescan changes margins or page order, old coordinates must not silently attach to the new file. Reprocess or retain a reference to the earlier authorized version. The digest helps identify content; it does not grant access. Every lookup of the original and its derivatives must still enforce the source's permissions and retention policy.

This record need not contain every intermediate image forever. A reproducible transform description and the retained source may be sufficient for some workflows. Others need the exact submitted raster to diagnose differences. Choose deliberately, accounting for storage, sensitive information, reproducibility and the consequences of an unavailable dependency.

Work backward through one crop-and-resize example

Suppose an already upright source scan measures 2,400 by 3,200 pixels. The application crops a region starting 400 pixels from the left and 600 from the top, with width 1,200 and height 1,600. It reduces that crop to 600 by 800 pixels for analysis. There is no padding, rotation or perspective correction in this example.

Assume the response contract supplies a relative rectangle with left 0.25, top 0.50, width 0.20 and height 0.10, measured against that analysis image. First expand the ratios there. Then undo the resize. Finally add the crop's offset. The sequence matters: adding an original-image offset before undoing scale would scale the offset too.

The intended source rectangle therefore runs from (700, 1400) to (940, 1560). Applying the relative values directly to the full original would instead produce left 600, top 1,600, width 480 and height 320. The result is a plausible rectangle on the right page, but it is not the same evidence region.

The following small JavaScript example uses continuous rectangle-edge coordinates. It illustrates this one transform only; it is not a general PDF or OCR adapter.

const crop = { x: 400, y: 600, width: 1200, height: 1600 };
const analysis = { width: 600, height: 800 };
const relative = { left: 0.25, top: 0.50, width: 0.20, height: 0.10 };
const scaleX = analysis.width / crop.width;
const scaleY = analysis.height / crop.height;
const source = {
  left: crop.x + relative.left * analysis.width / scaleX,
  top: crop.y + relative.top * analysis.height / scaleY,
  width: relative.width * analysis.width / scaleX,
  height: relative.height * analysis.height / scaleY,
};
// { left: 700, top: 1400, width: 240, height: 160 }

Round only at the final drawing boundary under a documented rule. Inverting the coordinate mapping does not recover detail lost by downsampling. Nor can it reveal text discarded by the crop. If an essential qualifier lies outside the submitted region, a perfectly mapped highlight still cannot make the answer complete.

Preserve polygons and use the library's page transform

For rotation, map every polygon vertex; do not merely move a rectangle's top-left corner and keep its width and height. A rotated quadrilateral may need a larger axis-aligned bounding rectangle, but that enclosing rectangle is less precise. Store the original polygon and distinguish it from a convenience box used for hit testing.

PyMuPDF's coordinate appendix explains the distinction between PDF and MuPDF coordinate systems, point units of one seventy-second of an inch, and transformation matrices and their inverses. This is why “PDF coordinates” alone is an inadequate adapter specification. Use the page and rendering library's documented transform rather than applying a universal vertical flip.

For a known invertible mapping, take coordinates backward through operations in reverse order. Record which direction each matrix maps. Reject singular or numerically unstable mappings instead of manufacturing a highlight. A perspective correction or nonlinear page dewarping needs the corresponding mapping; a single scale factor cannot describe it. If that mapping was not retained, show the analyzed derivative honestly or reprocess from a known frame.

Be careful at serialization boundaries too. The Google Document AI response guide documents normalized vertices and warns that zero-valued coordinate fields can be omitted from JSON. In that specific schema an omitted coordinate can represent zero. Teach the adapter that rule; do not globally convert all absent geometry to zero, and do not discard a valid corner because a truthiness test rejects zero.

The display layer introduces another mapping

After recovering source coordinates, the viewer still has to map them onto its rendered page. A fit-to-width image, a centered page with margins, a zoomed canvas and a cropped thumbnail have different mappings. Distinguish image pixels, canvas backing pixels and layout pixels. Keep the page image and its overlays under the same display transform.

For example, an uncropped 2,400-pixel-wide source shown at 600 layout pixels has a horizontal scale of one quarter. The worked example's left edge becomes 175 layout pixels from the displayed image's left edge, not from the outer panel. A centered image may have additional panel padding. Add that display offset once, not to the stored source location.

Right-to-left text does not reverse an image's coordinate axes. A Persian paragraph and a Latin identifier on the same scan still occupy the same physical page frame. Localize reading and interaction appropriately without reflecting every bounding polygon. Test mixed-script evidence, narrow viewports, zoom and page rotation as separate cases.

Correct geometry also does not settle context. Our guide to table values and header relationships explains why a highlighted number may need its heading, unit and note to support a claim. A citation can comprise several regions. Do not stretch one enormous box over unrelated material merely to make all the dependencies fit.

Make the fallback explicit instead of drawing a guess

Use separate states for extraction, location validation and interpretation. The following dispositions are our recommended integration policy, not features guaranteed by any provider.

A page-only reference is not an equivalent substitute when a downstream action requires exact field evidence. Set that rule with the workflow owner. A reviewer should see the limitation, not a reassuring highlight fabricated from approximate coordinates. Apply human-approval design to the consequential action separately from the evidence display.

Never reuse a highlighting rectangle as a redaction guarantee. Removing information from a document is a different operation with different verification requirements. This guide concerns finding and displaying evidence, not securely erasing it.

Test location errors without hiding them in text accuracy

Build fixtures with asymmetric, known landmarks so rotation and reflection cannot accidentally look correct. Include crop offsets, different horizontal and vertical scales, padding, right-angle rotations, skew, mixed page sizes, repeated text on several pages and a replaced file with the same name. Test both an untransformed baseline and each supported transform combination.

For the arithmetic example, assert the four source values and the two opposite corners. Add a forward-and-backward mapping test within a documented numerical tolerance. Round-trip success alone is insufficient: two mutually consistent but wrong transforms can cancel each other. Compare against independently known landmark locations and confirm document and page identity first.

Measure correct-source and correct-page rates separately from region overlap or corner distance. A large rectangle can obtain reassuring overlap while including the wrong field. Report whether the intended evidence is included, whether unrelated neighboring evidence is included, and whether the end-to-end claim is supported. Break results down by input format, preprocessing path, language and viewer version.

Track precise citations accepted, page-only fallbacks, mapping failures and reviewer corrections with their denominators. Revisit the fixtures whenever an extractor, image library, PDF renderer or viewer changes. The acceptance question is simple: can the team reproduce why this highlight points here, on this exact page version? If not, a less precise but truthful reference is better than a precise-looking mistake.

Source notes — reviewed October 7, 2026

The five primary references describe coordinate and library behavior, not comparative model performance. The worked example is synthetic; the architecture, tests and decision policy are ZharfAI's analysis. We have not validated these products on customer documents for this article.

• Amazon Textract BoundingBox — undated rolling API reference.
• Azure Document Intelligence response — v4 documentation, updated April 30, 2026.
• Pillow ImageOps.exif_transpose — documentation labeled 12.3.0, whose release navigation lists July 1, 2026.
• PyMuPDF coordinate appendix — documentation through 1.28.2, updated September 3, 2026.
• Google Document AI response handling — updated October 5, 2026.