Vision-Language Models can effortlessly recognize a dog in a park, transcribe a receipt, or write Python code from a whiteboard mockup. But feed them an HVAC equipment submittal or a commercial construction drawing, and their confidence turns hazardous.
In consumer document AI, a minor OCR typo is harmless. In heavy industry, construction, and mechanical engineering, a misread dimension chain or a hallucinated approval check can trigger millions of dollars in fabrication re-orders, structural failures, or contract disputes.
We ran an empirical evaluation of four flagship multimodal models across 48 real-world industrial drawing and equipment submittal runs. Here is what we found.
The Three Most Dangerous Failure Modes
When evaluating LLMs on technical drawings, measuring word error rate completely misses the point. The critical danger is relational hallucination—where every single word read is technically real, but combined into a catastrophic falsehood.
The Scope Inversion Failure
On architectural and demolition drawings, diagonal crosshatching universally signifies areas that are NOT part of the project scope. Multiple frontier models cleanly transcribed all the room names and dimensions inside the hatched zone, but failed to recognize the crosshatching—falsely asserting that demolition was required in those areas.
In practice, this commits a contractor to perform demolition work on property that is contractually excluded.
The Review Stamp Authority Fallacy
On an air-handling equipment submittal, the printed specification required maximum air velocity not to exceed 300 fpm. A red markup note from a reviewer proposed 400 fpm. Green approval checkmarks appeared on other sections of the document, but not on that requirement. When queried, models hallucinated that the checks meant the variance was formally approved and compliant.
Models cannot reliably distinguish between a printed requirement, a proposed revision, and an actual engineering sign-off.
The Schedule Flattening Trap
Engineering drawings are composed of multi-view projections alongside dense equipment schedules. Frontier models frequently collapsed 2D spatial layouts into a flat text stream, attributing a reception desk height to an HVAC unit, or swapping inlet dimensions with anchor positions.
Without 2D spatial anchoring, correct numbers get mapped to the wrong physical components.
Benchmark Scorecard
We scored all 48 model outputs across 6 core technical sheets. Models were evaluated on factual assertions (/60), spatial localization coordinates (/20), and relational engineering checks (/20):
| Model | Overall /100 | Factual Facts /60 | Relations /20 | Critical Errors |
|---|---|---|---|---|
 | 67.58 | 50.33 | 17.25 | 8 / 12 runs |
 | 54.00 | 41.00 | 13.00 | 8 / 12 runs |
 | 28.67 | 21.67 | 7.00 | 11 / 12 runs |
 | 28.25 | 21.00 | 7.25 | 12 / 12 runs |
Figure A. Mean scoring components across the four multimodal systems (Assertions /60, Relationships /10, Page checks /10).
Figure B. Paired score averages across individual drawing and submittal test documents.Model Breakdown: Where Each System Stumbled
Kimi K3
Impressive raw text and table extraction. It cleanly parsed multi-row chemical media calculation tables that tripped up other models. However, it exhibited severe over-confidence—authoritatively inventing curve identities and misinterpreting reviewer checkmarks as compliance approvals.
Qwen3-VL 235B
Strong comprehension of structured submittal tables, but suffered total collapse on dense construction drawings. On multi-elevation drawings, its responses repeatedly exceeded output token limits and hallucinated finish schedules to fill in the missing gaps.
Pixtral Large (Mistral)
Scored near-zero on dense construction drawings. While bounded submittal pages retained partial text, the model frequently hallucinated equipment schedules and corrupted dimensional values.
Amazon Nova Pro
Demonstrated partial text recovery on simple pages, but failed on multi-view sheets. Flagged frequent critical errors including false unit associations and invented equipment schedule identifiers.
The Engineering Solution: Deterministic Intermediate Representations
The lesson from this benchmark is clear: asking an end-to-end vision model to directly output business-critical conclusions from a raw drawing image is an architectural dead end.
Real engineering reliability requires a deterministic intermediate representation (IR) layer:
Building with Engineering Files?
Rine provides dependable intermediate representations for technical drawings and CAD documents.