Why we ran this benchmark
Across physical industries—architecture, construction (AEC), mechanical manufacturing, industrial automation, and energy systems—the world runs on technical documents and drawing sets. Decades into the digital era, multi-page PDFs remain the stubborn universal exchange format for construction bids, mechanical submittals, and fabrication specs.
As software teams build automated takeoff tools, estimating engines, and generative AI agents, they frequently turn to modern cloud extractors—including AWS Textract, AWS Bedrock Data Automation, Reducto, and Extend—to convert PDFs into structured data.
On the surface, modern multimodal OCR models look like magic: they parse characters, output clean Markdown tables, and identify paragraphs. But in physical engineering, character recognition is only a fraction of the challenge.
A number extracted in isolation is useless if a system cannot tell which view owns the dimension, whether an area is excluded by a hatching legend, or whether a red markup was an approval or a proposed change. To evaluate where extraction ends and true engineering understanding begins, Rine audited 48 real saved outputs against strict, multi-point ground-truth rubrics.
The short version
Across six selected engineering-document pages, four established extraction providers often recovered visible words, printed numerical specifications and useful page structure. The harder failures appeared when a downstream system needed to know what those facts meant: which drawing view owned a dimension, whether an area was excluded from scope, whether a red markup was a proposal or an approval, which curve belonged to which axis, or which footnote qualified a modeled result.
The audit contained 48 saved outputs: four providers × six pages × two input variants. All 48 were scored against a final page-specific rubric. None satisfied the entire clean-pass standard, even though six individual scores met or exceeded 90 out of 100. The distinction matters: this was not a finding that the vendors extracted no useful information. It was a finding that overall point totals did not guarantee complete preservation of the source relationships selected for evaluation.
A technically correct number can still be the wrong answer when it is attached to the wrong object, view, assumption or review status.
01 / What exactly did we test?
We selected six pages from two engineering-related PDF documents, not six independent full-document corpora. Two pages came from a construction drawing set; four came from an equipment submittal. The sample was intentionally varied: a demolition plan, an equipment schedule/elevation page, a marked-up specification, a mechanical assembly drawing, a performance chart and a table of operating assumptions. That diversity made it possible to examine distinct types of structure, but it does not make the sample representative of all engineering files.
Each page was evaluated twice: Track A used the original PDF page; Track B used an image-only version. The labels “original” and “image-only” are more accurate than a blanket claim of “selectable” versus “non-selectable”: some original pages were already mostly raster, and the equipment assembly drawing on page 29 had no useful ordinary text layer.
- Page 6: Demolition floor plans with keynotes & scope exclusions.
- Page 13: Equipment plans, 10-row schedule and interior elevations.
- Page 10: Specification & red-lined review variance marks.
- Page 29: Mechanical equipment assembly drawing (3 views).
- Page 53: Fan performance chart (twin y-axes).
- Page 84: Media-life calculation table with merged spans.
The final 100-point rubric assigned 60 points to page-specific factual assertions, 20 to source localization, 10 to relationships and 10 to page-specific checks. Its clean-pass rule was deliberately stricter than an aggregate score: an output had to score at least 90/100, earn full credit on every mandatory assertion, and contain no critical errors. A 94/100 result could therefore fail if it invented a review mark or failed an essential annotation-to-requirement association.
The adjudication covered 720 factual-assertion items, 480 localization items, 240 relationship items and 240 page-specific checks (1,680 total). The saved-output review recorded 647 scored deficiency rows, 58 curated finding records spanning 26 categories, and 29 critical-error scorecard records. Those are audit bookkeeping counts, not necessarily distinct independent errors: one underlying mistake can affect multiple assertions or appear in multiple records.
02 / The results in numbers
Across the 12 saved outputs per provider, Extend had the highest average adjudicated score, followed by Textract, Reducto and Bedrock Data Automation. This is a ranking within this selected six-page audit, not a provider-wide accuracy benchmark or a production procurement recommendation.
Mean adjudicated scores across 12 saved outputs per provider
Figure 1. Equal-weight mean rubric scores across six pages and both variants. Units are audit points out of 100—not calibrated percentages of correct extraction.| Provider | Track A mean | Track B mean | Overall | ≥90 scores |
|---|---|---|---|---|
| Extend | 80.50 | 80.33 | 80.42 | 4 |
| AWS Textract | 77.50 | 77.67 | 77.58 | 2 |
| Reducto | 76.83 | 76.50 | 76.67 | 0 |
| AWS Bedrock Data Automation | 71.00 | 73.17 | 72.08 | 0 |
| Page and task | Textract | Bedrock | Reducto | Extend |
|---|---|---|---|---|
| Drawing p6Demolition floor plans | 74 / 70 | 69 / 69 | 81 / 82 | 83 / 83 |
| Drawing p13Equipment schedule & interior elevations | 58 / 67 | 49 / 66 | 61 / 67 | 73 / 60 |
| Submittal p10Specification & review marks | 91 / 87 | 89 / 85 | 86 / 86 | 94 / 94 |
| Submittal p29Equipment assembly drawing | 78 / 76 | 69 / 69 | 66 / 62 | 67 / 76 |
| Submittal p53Fan performance chart | 75 / 75 | 72 / 72 | 82 / 77 | 73 / 76 |
| Submittal p84Media-life calculation table | 89 / 91 | 78 / 78 | 85 / 85 | 93 / 93 |
The highest mean across eight outputs occurred on submittal page 10, the marked-up specification (89.00). The lowest was construction drawing page 13, with schedules and interior elevations (62.63). Page 84—the media-life table—also scored comparatively well (86.50) while remaining vulnerable to relationships and footnote binding. The gap between a high mean and a usable structured record is the central finding of this evaluation.
03 / Six pages, six different ways meaning can break
Demolition drawings: when “not in scope” becomes “in scope”
The demolition-plan legend contained a consequential exclusion: cross-hatched areas were “NOT PART OF SCOPE.” Both Textract variants and both Bedrock variants lost or damaged that negation. Elsewhere in the page, providers often located room identifiers, keynotes and dimension strings, but that did not ensure correct interpretation of the graphical conventions. In one saved Textract image-only output, an instruction for 48 hours of notice appeared as 8 hours.
The full horizontal dimension chain and separate vertical dimension were never fully associated with their correct roles in any of the eight outputs. This is a useful distinction for engineering software: a system can extract the value 22′ 11″ and still fail if it cannot establish what the measurement applies to.
Equipment schedules and elevations: exact identifiers are fragile
The second drawing page contained a ten-row equipment schedule plus plan and elevation details. A saved Extend image-only output changed refrigerator model AL57G to AL5TG and responsibility code CFCI to OFCI; a Reducto original-page output altered a microwave model. Textract merged two equipment rows in one result. These are not merely spelling differences when the intended downstream output is a typed equipment record.
The same page illustrated geometric-role confusion. A 3′ 1″ depth in a reception-desk plan was described as a height. Nearby views of a standard table and an accessible table could share visible features while referring to different dimensions and finishes. Every output missed full credit on a mandatory reception-section dimension/role association. This page produced the lowest mean score of the six tasks (62.63).
Marked-up specifications: proposed change is not approval
Submittal page 10 juxtaposed a printed B2 upper limit—filter-face velocity “shall not exceed 300 fpm”—with a red review note recommending 400 fpm. These are two different kinds of source information. A safe system must preserve the printed requirement, the separately added recommendation, and what the page does or does not prove about disposition. No approval of a 400-fpm variance was established in the audited source.
All eight saved outputs recovered both printed numbers, but all eight earned only partial credit for the mandatory relation among the note, its red rectangle and the B2 requirement. Twelve green check marks appeared elsewhere on the page, and none was on B2. Both Reducto outputs inserted a checked B2, a fabricated review mark. Bedrock, by contrast, preserved all 12 actual green-check crop assets and their positions. Extend scored 94/100 on both tracks, yet the key annotation relation was incomplete.
Mechanical assembly drawing: a dimension needs a view and a role
Submittal page 29 depicted an equipment assembly in top, side and front views. A body dimension of 83.09 [2110.41] was distinct from the explicitly marked overall height of 90.03 [2286.69]. None of the eight outputs fully preserved that mandatory distinction. A minimum service clearance of 30.00 [762.00] also had to remain a clearance, not become another dimension of the housing.
The plan included six stage bubbles in a nontrivial spatial sequence, with several leaders connecting stage labels to processing passes. All eight outputs only partially captured that mapping. Some anchors and “DO NOT SCALE” instructions were recognized across providers, demonstrating why it is misleading to describe a page simply as successful or failed. The task is not just reading labels; it is binding labels, leaders, dimensions and viewpoints to the correct source objects.
Fan performance chart: printed selections are not the same as curve readings
Submittal page 53 included both a printed fan-selection block and a chart with separate axes: flow in ACFM along x, static pressure against the left y-axis, and input power against the right y-axis. The source printed 11,000 ACFM, 6 in. wg, 1,626 rpm and 14.0 bhp. All eight outputs substantially recovered the printed selection facts and qualifiers. That positive performance should not be confused with extracting new numerical values from the plotted curves.
At roughly 13,000 ACFM, the frozen visual evaluation key accepted approximate readings of 3.8–4.2 in. wg for the green pressure trace and 12.4–13.2 bhp for the blue power trace. All eight outputs earned zero on the specific assertion requiring both approximations. Two Extend outputs also called an unlabeled gray line the “system curve,” assigning meaning not supplied by the printed figure.
Media-life table: the numbers survived, but their qualifications sometimes did not
Submittal page 84 listed flow and media properties alongside anticipated media life. The printed results included 11,000 cfm, 2,254 lb of combined media, and anticipated life of 3.1 months / 0.3 years. These values were broadly recovered. But the names “Sulphasorb XL” and “CPS12” each applied across four associated rows, and a flattened output could lose that merged-cell relationship while preserving most individual numbers.
The two life predictions also carried two footnotes: operation at 24/7/365, and “Media 1 only in calculation.” These qualifications apply to both predicted-life rows; omitting them changes the claim being made. Bedrock uniquely achieved full credit in both tracks on the merged Media 1 association, while Reducto and Extend did better on the life-row footnotes. Extend still scored 93/100 in both tracks but missed required merged-span fidelity.
04 / Provider differences were real—but task-dependent
The selected pages did not produce a clean universal winner. The provider with the highest mean was not uniformly best on specific assertions, and the correct choice for a real product would depend on what information the workflow actually consumes.
Its saved outputs frequently supplied fine-grained native text and location anchors, with useful exact numbers and local source strings. That was evident in the annotated-specification page, although only a subset of the 12 green checks was fully identified. Limitations included the demolition-scope negation, equipment-row merging, incomplete dimension associations and the 48-hour-to-8-hour error in one image-only result.
Bedrock preserved all 12 visible check-mark crops on the marked-up specification and correctly represented the full four-row Media 1 name span in both table variants—two distinctive positives. Its coarser location boxes sometimes lost credit in the final localization rubric. It also suffered negation and source-identity mistakes in drawing pages.
Reducto performed relatively strongly on higher-level text blocks and on the fan-chart page, reaching 82/100 for the original chart track. It also handled the four-row Media 2 association and some life-table footnote relationships well. But both marked-up-specification outputs invented a check at B2, a safety-relevant source contradiction.
Extend returned the highest mean in this sample (80.42/100), including 94/100 on both marked-up-specification variants and 93/100 on both media-life variants. It also produced good fine-grained source boxes. Yet it accumulated 14 critical-error scorecard records among its 12 outputs—the largest provider tally in this audit—including unsupported captions, drawing-role substitutions and chart-curve labeling.
This combination is why “highest average wins” is not a valid selection rule for a safety- or fidelity-sensitive downstream task.
05 / Did rasterizing the documents help? Not consistently
Track B did not move all providers or pages in the same direction. Bedrock’s drawing-page-13 total increased from 49 on the original to 66 on the image-only variant, while Extend fell from 73 to 60 on the same page. Textract improved from 89 to 91 on media-life page 84, yet both outputs missed mandatory span/qualifier requirements. Several pairs were unchanged. The means for each provider barely moved across variants, except for a modest upward difference for Bedrock in this tiny sample.
These differences are observations of individual saved responses, not replicated randomized trials. Because original pages were not uniformly text-selectable, and the historical exact input bytes and provider settings could not be independently verified, they do not justify a blanket claim that image-only conversion improves or worsens engineering-document extraction.
06 / How far can we trust the benchmark?
The study was a retrospective adjudication of saved outputs with human-reviewed final judgments. It was not a fresh provider API re-run, independent repeated trial, or time-and-cost comparison under tightly controlled deployment settings. The final scorecards, answer keys, run registry and adjudication records were preserved, but the historical archive did not include all 48 unmodified provider-native responses, the 12 one-page input variants or the two full source PDFs. The original source-tree files were inventoried during the audit, but their complete underlying bytes have not been transferred to the external archive as described in the inventory.
The final answer keys were versioned and used in adjudication; the audit record does not independently prove the exact same keys were fixed before the original provider runs. We therefore treat the measurements as selected-page, saved-output review results. We do not represent them as independent blind evaluation, statistical inference about average vendor performance or estimates of deployed production reliability.
- Which provider would win on an unseen customer corpus, under another configuration or for an output contract narrower than these rubrics.
- Whether all four systems would show the same failure modes with different prompts, product versions, preprocessing or additional validation.
- A fair cross-provider latency or cost ranking: recorded response timing fields had different clock scopes, while billed costs, retries and end-to-end economics were unverified.
- That any of these weaknesses is an unmet market demand for a new Rine API—or that Rine has already built a production-ready solution to them.
- That a reported clean-pass failure is equivalent to an unusable result. Several providers extracted useful content and some passed difficult individual assertions.
07 / A better question for engineering-document products
A generic “parse this PDF” request hides the actual acceptance criteria. A procurement or submittal application may care about product ID, required velocity, reviewed alternative and disposition. A maintenance system may need a table record with its units and forecast assumptions. A document search tool may only need retrievable passages and source-page citations. A planning system may care about drawing-view dimensions; another application may not need geometry at all. The same saved response can be useful for one product and unacceptable for another.
Our immediate conclusion is therefore about evaluation design, not product architecture: benchmark the job a customer needs done, not simply whether a vendor returns lots of plausible text. Define the expected record or operation, its mandatory associations, the cost of a false assertion, and the evidence the system must carry so the customer can inspect or reject a conclusion.
- Downstream consumption: What exact fields, relationships and downstream actions will consume the output?
- Factual grounding: Which values must be copied exactly, and which are inferences or estimates?
- Provenance: What source coordinates, page regions, marks or annotations must accompany each claim?
- Critical errors: What errors are critical enough to require abstention or human review rather than a plausible answer?
- Integration threshold: What accuracy, cost, review effort and repeatability would make the operation worth integrating?
These questions are also Rine’s next research boundary. The October 25 audit is an internal incumbent baseline on Rine-selected pages, not an acceptance test on documents supplied by external customer workloads. Product direction should follow representative customer inputs, their exact requested outputs and acceptance criteria, then a controlled evaluation of the best available primitives. Where existing tools satisfy that contract, the infrastructure should use them. Where repeatable gaps remain and customers are prepared to depend on an external component, that becomes a candidate for differentiated Rine engineering.
Conclusion / Readability is not reliability
The most useful result of this 48-output experiment is not that engineering PDFs are impossible, and not that one provider “won.” It is that extraction quality changes depending on what we count as correct. Across these pages, the difficult mistakes were not always missing strings: they were missing negations, misplaced identifiers, unbound dimensions, invented review status, unsupported chart labels and detached assumptions. All are cases in which text that appears plausible can become a false structured record.
A reliable engineering-document workflow must make its output contract explicit. It should distinguish source text from interpretation, preserve source-to-claim relationships, expose uncertainty and allow important errors to be rejected. Whether any particular workflow needs more than an existing provider already offers is a question for representative customer-file tests—not an inference from 0/48 clean passes alone.
The goal is not to extract more text. It is to preserve the meaning a downstream engineering workflow is entitled to rely on.
Study unit: six PDF pages from two source documents, four existing extraction providers, two variants per page, one saved output per provider/page/variant combination. Date of completed audit: 25 October 2026. Scoring: final 100-point page-specific rubric; clean pass requires score ≥90, all mandatory assertions at full credit and no critical errors. Provider performance scores are descriptive review points, not percentages of correctly extracted information.
Primary internal artifacts: Rine — Round 1 — Living Page-by-Page Audit Findings (25 October 2026, revised 26 October); Rine — Round 1 — Comprehensive 48-Run Audit (clean edition, 26 October 2026); Rine-Round1-Completed-Audit.zip with frozen answer keys, 48 evaluator scorecards, 1,680-item adjudication ledger and error/finding registries. Full evidence files, historical scorecards, and audit logs are cataloged in Rine's internal research archives.
Editorial note: this document is an editorial technical dispatch, not an announcement of a current commercial Rine API capability. Private source figures and raw vendor response dumps are intentionally not reproduced. Public release has been reviewed for source permissions and representation of vendor names.
Building software around complex engineering drawings, specifications, or technical documents? Share your representative workload with our research team.