If you are an engineering leader building AI workflows for industrial files, you currently have two choices: stitch together dedicated document extraction APIs, or feed high-resolution page renders into frontier vision models.
Both approaches promise effortless automation. But when subjected to rigorous evaluation against real commercial blueprints and complex equipment schedules, both approaches hit a wall—for opposite reasons:
Great at finding where words live on a page (bounding box coordinates), but completely blind to relational engineering context (which dimension belongs to which elevation).
Great at synthesizing complex tables into fluent prose, but completely unanchored to page coordinates and prone to confident, catastrophic hallucinations.
The 96-Output Performance Matrix
We scored 96 saved runs across 8 systems on 6 challenging industrial sheets. Every output was adjudicated on factual extraction (/60), spatial coordinate localization (/20), and relational engineering checks (/20):
| System | Overall /100 | Facts /60 | Coords /20 | Relations /20 | Critical Errors |
|---|---|---|---|---|---|
 | 80.42 | 46.17 | 18.25 | 16.00 | 7 / 12 runs |
 | 77.58 | 46.17 | 17.17 | 14.25 | 2 / 12 runs |
 | 76.67 | 47.00 | 15.08 | 14.59 | 6 / 12 runs |
 | 72.08 | 45.17 | 12.50 | 14.41 | 2 / 12 runs |
 | 67.58 | 50.33 | 0.00 | 17.25 | 8 / 12 runs |
 | 54.00 | 41.00 | 0.00 | 13.00 | 8 / 12 runs |
 | 28.67 | 21.67 | 0.00 | 7.00 | 11 / 12 runs |
 | 28.25 | 21.00 | 0.00 | 7.25 | 12 / 12 runs |
Complete score heatmap across 12 test runs per system. Darker colors represent near-zero scores on dense drawings.Why Does Extend Lead Overall While Kimi Leads Facts?
Looking only at the composite score (Extend at 80.42 vs Kimi at 67.58) creates a misleading impression. When we peel back the score components, we see fundamentally different design philosophies:
Figure A. Score decomposition: Specialized engines earn 12–18 points on spatial localization, whereas multimodal AI scores 0 on coordinates but higher on raw factual assertions.The Localization Penalty
Specialized document APIs output bounding box polygons for every detected word, earning them 15 to 18 points on coordinate localization. Multimodal models generate natural prose answers without bounding boxes—automatically zeroing them out on 20% of the rubric.
If you evaluate purely on non-spatial factual comprehension (out of 80), Kimi K3 takes the lead at 67.58 / 80, outperforming Extend (62.17) and Reducto (61.58).
Hallucination Risk: Conservative OCR vs Confident AI
The most important finding for engineering workflows is the distribution of critical errors:
Figure B. Number of runs with critical hallucinations. AWS Textract and Bedrock proved the most conservative, while generative multimodal models confabulated frequently.AWS Textract and Bedrock Data Automation triggered critical errors in only 2 of 12 runs. Because they do not invent semantic narrative, they rarely confabulate non-existent specifications.
Kimi (8/12), Qwen (8/12), Nova Pro (11/12), and Pixtral (12/12) repeatedly invented technical conclusions—naming unlabeled curves, hallucinating review approvals, and transposing dimension chains.
The Architectural Conclusion: Why Both Sides Need an Intermediate Layer
This benchmark proves that neither specialized OCR APIs nor raw multimodal LLMs can autonomously parse engineering documents on their own:
- Specialized OCR gives you coordinates, but lacks the spatial intelligence to associate leaders, equipment tags, and multidimensional schedules.
- Multimodal LLMs give you semantic fluency, but lack coordinate verification and hallucinate dangerous conclusions when visual data is ambiguous.
The only robust path forward is a hybrid architecture: parsing raw geometric vectors and coordinate bounding anchors into a deterministic Intermediate Representation (IR), and enforcing strict schema validation before allowing AI models to draw semantic conclusions.
Building with Complex Technical Files?
Rine provides dependable intermediate representations designed specifically for CAD, BIM, and engineering drawings.