Repro: macOS 27 Vision OCR (VNRecognizeTextRequest) garbles mixed Chinese/English text; only the first recognition language is honored - missuo/macos27-vision-ocr-bug

_Originally published on hexisteme notes._ I had a 27B vision model running locally (IQ4_XS quantized, 15GB resident) and needed to decide whether it was worth using for OCR. The comparison was macOS's built-in Vision framework (`VNRecognizeTextRequest`) — a **dedicated** text-recognition engine, free, zero memory footprint. My expectation going in: the specialist wins on character accuracy, and the general-purpose model is reserved for when you need semantic understanding. Slow and expensive, use sparingly. That expectation was wrong, and it was wrong in a way that would have been invisible in production. ## Method: an image whose answer I already knew The usual mistake in an OCR comparison is measuring against real documents, where you don't have ground truth. Then you can't distinguish _plausible_ output from _correct_ output — and plausible output is exactly what both engines produce when they fail. So I rendered a 1100×720 test image with the answer fixed in advance: * A title and date in Korean, a 4-column × 3-row table (model / memory / speed / status), four lines of prose * One adversarial line: `A0-1lO9-B8` — digit `1` next to lowercase `l`, capital `O` next to digit `0` * Two empty table cells containing `-` Then I looked at it. The first render was **wrong** — a label came out as tofu boxes (□□), because the monospace font had no Korean glyphs. If ground truth is broken at the moment you fix it, every measurement afterwards is void. That check costs thirty seconds and it's the whole experiment. ## The results | Local 27B VLM | Apple Vision (dedicated) ---|---|--- Character errors | **2** | **8** Reading order | preserved | **destroyed** Table cells dropped | 0 | 2 (the `-` cells) Wall clock | 82.8s (cold) | **0.27s** Vision's eight errors: `IQ4_XS`→`I04_XS`, `cloud`→`cLoud` (twice), `tok/s`→`tok/5`, two characters inside a code string, em dash `—`→`-`, arrow `→`→`->`. **300× faster.** On accuracy alone, four times the error rate on a document of this size is arguably a fine trade. Accuracy alone is not what decided it. ## The difference was structural, not lexical Vision returned the table **decomposed by column.** Three model names in a row, then three memory figures, then the speed and status columns appended at the end of the document. Which means: **you cannot recover which speed belongs to which model from the output.** The row associations are gone. Not garbled — _gone_. The characters are all there, correctly grouped, in a well-formed sequence, and the relation between them has evaporated. The VLM kept the rows. Next to that, 2 errors versus 8 is a rounding difference. What this actually says is narrower than "the general model is better": > **"A dedicated tool beats a general one" depends entirely on where you cut the task.** Vision is dedicated to _character recognition_. It is not dedicated to _document understanding_. My task needed the second and I was picking tools by the first one's benchmark. The specialist was genuinely better at the thing it specializes in — I had just mislabeled what I needed. ## Why this breaks the cheap-first fallback The obvious architecture is: run the cheap engine, detect failure, escalate to the expensive one. Almost everyone reaches for this. It requires failure to be **detectable**. Column-shredded output is syntactically perfect. It has plausible text, plausible structure, no error signal of any kind. Downstream, it is indistinguishable from a correct read. The information didn't get corrupted — it got _dropped_ , and dropped information leaves no residue to detect. This generalizes past OCR. Any escalation ladder — cheap model then expensive model, cache then origin, heuristic then solver — is only sound when the cheap tier's failure mode is **observable at the boundary**. If the cheap tier can fail by silently discarding a relation rather than producing a wrong value, "cheap first" isn't an optimization. It's an undetected data loss path with a cost saving attached. ## Both engines failed in exactly the same place The adversarial string `A0-1lO9-B8`: Attempt | Output ---|--- VLM, full image | `A0-1109-B8` — 2 misreads VLM, that line at 4× with an explicit "distinguish 0/O and 1/l" instruction | `A0-1l09-B8` — recovered `l`, still lost `O` Vision, on the enlarged crop | `A0-1109-B8` — 2 misreads, unchanged I inspected the enlarged image myself. The font renders digit `0` with a slash through it and capital `O` as a plain oval. The two glyphs are **visibly different.** This isn't image ambiguity that more pixels would resolve — it's both engines genuinely misreading a distinguishable character, and resolution doesn't touch it. So: > **No OCR engine can be trusted on strings where homoglyphs change the meaning** — codes, IDs, > hashes, addresses, license keys. That's not a tool-selection problem. It's a property of the entire tool class, which means the remedy isn't a better engine. It's human confirmation or a checksum. If you're about to build an OCR path for identifiers, build the checksum first. ## What I adopted Vision as the first pass; escalate to the VLM only for documents where reading order carries meaning — tables, forms. **With one correction to that rule, from the paragraph above:** since column-shredding isn't detectable downstream, "escalate on failure" doesn't work for tables. If the corpus is mostly tables, go to the VLM _first_ and eat the 300×. Cheap-first is only valid when failure is visible. ## What would change my mind * If the target documents are mostly prose, Vision alone is sufficient and the VLM is a 300× waste. * If Vision gains layout analysis and starts preserving table structure, this verdict is dead. * The homoglyph failure is common to both engines, so improving one doesn't touch that part of the conclusion. _More notes at hexisteme.github.io/notes._

什么是 OCR Server OCR Server 是一款基于 Apple Vision Framework 的 iOS 应用，可将 iPhone 变成局域网内可访问的本地 OCR 服务器。它提供网页上传与 JSON API，支持多语言自动检测与高速识别，所有处理在设备端完成，无需依赖云端，数据不出本机。 OCR Server核心能力 高精度文字识别：基于 Vision 的 VNRecognizeTextRequest，对图像进行文字定位与提取，适用于常见场景的文本抽取。 多语言自动检测：自动识别多语种内容，减少手动切换语言的成本。 网页与 API：在同一网络内，通过应用显示的 IP 用浏览器上传图片并获取识别结果；同时提供返回 JSON 的 Web API，便于系统集成与自动化。 边界框与结构化结果：新版在 JSON 中加入文字位置等信息，便于二次标注与可视化。 100% 本地处理与隐私保护：处理过程在 iPhone 上完成，不上传云端，适合对合规与保密有要求的场景。 适用场景 开发测试：为移动或后端应用提供局域网 OCR 测试环境。 团队协作：在办公室/工作室内共享 OCR 服务，统一入口、减少账号与配额管理。 离线处理：网络受限或禁止外连的环境下完成文字识别。 批量/集群：用多台 iPhone 组成轻量 OCR 集群，提升吞吐。 OCR Server如何使用 基本步骤 打开应用，内置服务器自动启动。 在同一 Wi-Fi/局域网内，用任意设备访问屏幕上显示的 IP。 通过网页上传图片，数秒内返回识别文本与结构化结果；或以 API 方式对接业务。 稳定运行建议 需要长时间不间断服务时，可启用 iOS 引导式访问（Guided Access），锁定在当前应用并管理自动锁屏行为，减少误触与待机中断。设置路径见 Apple 支持文档。 OCR Server下载地址 GitHub地址： App Store下载地址：OCR Server

WWDC23が開催されたのに合わせ、Appleの国際化担当シニアソフトウェアエンジニアリングマネージャーKaran Miśra氏が、iOS 17、iPadOS 17から日本語手書きキ...