TikTok video by Katarzyna

My little iOS app PictureFramer does one thing: you photograph a framed painting in a museum, and it straightens the photo — finds the frame, fixes the perspective, keeps a clean strip of wall around ...

Until now, digitizing a framed painting with PictureFramer meant a three-app dance: open Camera, shoot the picture, switch to Photos to check it, then open PictureFramer and re-find the shot in the pi...

Until now, digitizing a framed painting with PictureFramer meant a three-app dance: open **Camera** , shoot the picture, switch to **Photos** to check it, then open **PictureFramer** and re-find the shot in the picker. Version 1.4.0 collapses that into one flow: tap **Take Photo** inside the app, shoot, and land directly in the editor with the artwork already detected and perspective-corrected. This post walks through how the feature is built — and why the interesting part is how little code it took. ## The user-facing change On the picker screen there is now a **Take Photo** button next to the familiar **Choose Photo** one. It opens the standard iOS camera UI — shutter, flash control, and the retake/use-photo confirmation you already know. The moment you tap _Use Photo_ , PictureFramer's pipeline takes over: Vision detects the outer edge of the frame, Core Image straightens the perspective, and you're in the editor adjusting corners and margin, exactly as if you had picked the shot from your library. One deliberate difference from shooting with the Camera app: **the raw capture is never saved**. The skewed, uncorrected original exists only in memory. The only image that ever reaches your photo library is the final, straightened export — no cluttering your camera roll with throwaway shots taken at an angle. ## Design constraint: the pipeline must not care PictureFramer's core (`Sources/Core/`) is a UI-free, fully unit-tested pipeline: detect → margin → correct. Its input is a `CGImage` in a canonical coordinate space; it has no idea whether the pixels came from the photo library, and it shouldn't start caring now. So the design goal was: **zero changes to`Sources/Core/`**. The camera is just another source of bytes. The one refactor that made this true lives in the view model. Loading a library photo used to be a single method that did two jobs: resolve the `PhotosPickerItem` to `Data`, then decode, downscale, and run detection. Splitting it gave us a shared funnel: /// Shared load funnel for both photo-picker and camera captures. func load(data: Data) async { stage = .loading errorMessage = nil guard let image = Self.normalizedCGImage(from: data) else { errorMessage = "Couldn't load that photo." stage = .picking return } sourceImage = image let base = await Task.detached(priority: .userInitiated) { downscaled(image, maxDimension: 1600) }.value previewBase = base previewScale = CGFloat(base.width) / CGFloat(image.width) await runDetection() } `load(item:)` (the photo-picker path) is now a thin wrapper that resolves the picker item to `Data` and calls `load(data:)`. The camera path calls `load(data:)` directly. Both sources converge before any real work happens, so every downstream behavior — detection, error handling, stage transitions — is shared and tested once. ## The camera wrapper: dumb glue on purpose We considered a custom `AVCaptureSession` viewfinder with a live Vision rectangle overlay — see the frame outlined in real time, shoot when it locks on. Tempting, and explicitly rejected. It would mean hundreds of lines of session management, preview-layer coordinate math, and a second rectangle-detection code path to keep in sync with the real one. The system camera UI already does everything the flow needs. So `CameraPicker` is a `UIViewControllerRepresentable` wrapping plain `UIImagePickerController`, and it is intentionally logic-free: func imagePickerController( _ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey: Any] ) { // 0.95 keeps the EXIF orientation tag and near-lossless pixels; // normalizedCGImage(from:) bakes the orientation in downstream. if let image = info[.originalImage] as? UIImage, let data = image.jpegData(compressionQuality: 0.95) { parent.onCapture(data) } parent.dismiss() } That's essentially the whole file: read the confirmed capture, encode it as near-lossless JPEG, hand the bytes to a callback, dismiss. Cancel just dismisses. Because there is no logic here, there is nothing here that needs unit tests — a property we wanted, not an accident (more below). Two presentation details worth knowing if you build something similar: * The camera is presented via `fullScreenCover`, not a sheet. Sheet-presented camera pickers are known-glitchy (broken layout, dead shutter on some iOS versions). * The button only renders when `UIImagePickerController.isSourceTypeAvailable(.camera)` is true — so it disappears automatically in the simulator instead of crashing on tap. ## Orientation: the bug that never happened Camera captures are the classic source of EXIF-orientation bugs — shoot in portrait, get a sideways image, start sprinkling rotation fixes through the pipeline. PictureFramer's architecture made this a non-event. The app's load-bearing invariant is that its canonical coordinate space (full-resolution source pixels, lower-left origin) **never sees orientation** : `normalizedCGImage(from:)` bakes the EXIF orientation into the pixels at decode time, before anything else runs. The camera path feeds JPEG data — with its orientation tag preserved by `jpegData(compressionQuality:)` — into that same decoder. Portrait, landscape, upside-down: handled with zero new code, because the invariant was already paying rent. ## Permissions Unlike the photo picker (which runs out-of-process and needs no permission string), the camera requires `NSCameraUsageDescription` and runtime authorization. The flow checks `AVCaptureDevice.authorizationStatus(for: .video)` on tap: * `.denied` / `.restricted` → don't present the camera; show an inline error with a link to the app's Settings page, mirroring the existing pattern for photo-library export denial. * Anything else → present; iOS shows its own permission prompt on first use. ## Testing where the logic lives The test strategy follows directly from the "dumb glue" split: * **Unit tests (Swift Testing)** drive `load(data:)` directly — no camera, no picker. A JPEG-encoded fixture image (drawn headlessly into a `CGBitmapContext`, no bundled assets) must reach the adjusting stage with a detected quad; garbage bytes must produce the "Couldn't load that photo." error and return to the picker. Since both input sources funnel through this method, these tests cover the camera path's entire logic. * **The wrapper is not unit tested.** It contains no branches worth testing, by design. * **No XCUITest** for the capture flow — the simulator has no camera, full stop. The end-to-end flow was verified manually on hardware via TestFlight. Pushing all logic into a testable, source-agnostic method and keeping the UIKit adapter logic-free is the whole trick. When the untestable part has nothing in it, "we can't test the camera in CI" stops being a coverage hole. ## Takeaways * **New input source ≠ new pipeline.** Converge sources on a shared funnel (`load(data:)`) as early as possible; everything downstream stays written and tested once. * **The system camera UI is usually enough.** A custom `AVCaptureSession` viewfinder is a big maintenance surface; reach for it only when you truly need live overlays. * **Make the untestable layer logic-free** , then don't test it. Test the funnel it feeds instead. * **Bake EXIF orientation in at decode time** , once, and orientation bugs stop existing as a category. The net cost of the feature: one 65-line wrapper, a ~30-line view-model refactor, a button, and a permission string. PictureFramer 1.4.0 is live now.

A museum-photo straightener built on Vision and Core Image, with optional bring-your-own-key reflection removal — free, iOS 17+, no data collected. Download on the App Store · pictureframer.corti.com...

**A museum-photo straightener built on Vision and Core Image, with optional bring-your-own-key reflection removal — free, iOS 17+, no data collected.** Download on the App Store · pictureframer.corti.com PictureFramer solves one narrow problem properly: you photograph a framed painting in a museum, you can never stand dead center, and every shot comes back rotated and keystoned with the frame converging toward one side. Cropping does not fix perspective, and document scanners crop _to_ the detected edge — which throws away the frame and the wall, the two things that make a picture of a painting look like a catalog plate instead of a snapshot. The app imports a photo, finds the outer edge of the artwork, corrects rotation and keystone in a single transform, keeps a configurable margin of _real wall pixels_ around the frame, and writes the result back to the photo library at full resolution. After a TestFlight preview, it is now generally available. ## The architectural decision that carried the project iOS image pipelines juggle at least three coordinate systems: Vision returns normalized coordinates with a lower-left origin, Core Image works in pixels with a lower-left origin, and UIKit/SwiftUI draw from the top-left. Most bugs in this class of app are silent flips and scale confusions between them. PictureFramer declares one canonical space up front — **full-resolution source-image pixels, lower-left origin** — chosen to be identical to Core Image's space. Everything else is defined relative to it: * **Vision → canonical is a pure scale.** No flip, because both spaces are lower-left. A single function, `VisionQuadConversion`, is the only code permitted to interpret Vision's normalized output. * **Canonical →`CIPerspectiveCorrection` is the identity.** Detected corners pass straight into the filter as `CIVector`s. * **Exactly one y-flip exists in the app** , inside `DisplayMapper`, at the SwiftUI boundary. It maps canonical pixels to the aspect-fitted display points and back, and every gesture — corner drags, preview panning — routes through it. A `Quad` value type (four `CGPoint`s in canonical space) is the single currency of the pipeline. Detection may run on a downscaled copy for speed, but the detector converts to full-resolution pixels _before returning_ , so "which scale is this quad in?" is not a question the rest of the codebase can ask. ## Detection `VNDetectRectanglesRequest` does the primary work, tuned for the actual case — a large framed rectangle filling most of the photo — with a high minimum size and a wide aspect-ratio range. When that returns nothing (small artworks, extreme panoramas, low-contrast frames), a second pass runs with permissive thresholds. Observations are ranked by confidence with area as the tie-breaker, so the outer frame edge wins over an inner mat edge. On a set of eight real handheld museum photos, the default configuration detected 8/8, including an unframed canvas where the stretcher edge was sufficient. When detection does fail, the editor falls back to a centered draggable quad, so the flow never dead-ends. ## Margin is wall, not padding The margin has to be applied _before_ perspective correction, in source space, or it is synthetic border fill rather than the actual wall. Expanding a tilted quad is not a negative inset. Each edge is offset outward along its outward normal (computed against the centroid, so winding order is irrelevant), and adjacent offset edge lines are re-intersected to produce the new corners. The expanded quad then samples real background pixels through the same homography as the painting. Two edge cases turned up in testing: * **Oversized margins** — more margin requested than wall available — clamp per corner to the image bounds, degrading to "everything up to the photo's edge." * **Negative margins** can collapse the quad through zero and flip its winding. A naive convexity test still passes in that state, because all the cross products change sign together. A shoelace signed-area check comparing winding before and after expansion catches it; a unit test caught it before any user did. One calibration surprise worth knowing: `CIPerspectiveCorrection` does not size its output from the quad's edge lengths. It reconstructs the rectangle's true proportions via the homography, so a keystoned quad can produce an output ~25% taller than its average edge length. ## Optional: reflection removal through museum glass Straightening cannot help with skylight streaks, spotlight bloom, or a green exit sign glowing in the varnish. Reconstructing what is underneath means _inventing_ pixels, which means a generative model — and generative models have a habit of improving things you did not ask them to touch. For 130-year-old brushwork, that is disqualifying. So the feature is built around a client-enforced invariant: **every pixel outside the user's mask is bit-identical to the original.** Not visually identical. The flow that guarantees it: 1. The user paints the glare, producing a grayscale mask (white = repaint). 2. The app crops a padded bounding box around the mask, resizes to the provider's upload size, and sends only that crop plus the mask. 3. The returned patch is resized back and composited into the full-resolution image through a Core Graphics clip mask — `CGContext.clip(to:mask:)` with the original drawn first. Where the mask is black, the framebuffer keeps the original bytes. No Core Image, no color-managed round trip, no drift. 4. The mask edge gets a Gaussian feather, multiplied by the binary mask first so softness only ever grows _inward_. A unit test iterates every pixel of a fixture and asserts exact equality outside the mask. Two providers sit behind a four-line protocol: protocol InpaintingProvider: Sendable { func uploadSize(for cropSize: CGSize) -> CGSize func inpaint(image: CGImage, mask: CGImage, apiKey: String) async throws -> CGImage } **OpenAI (`gpt-image-1`)** exposes a real inpainting endpoint: `images/edits` takes an image plus a mask in which _transparent_ pixels mark the repaint region, so white-means-repaint grayscale is converted with alpha = 255 − gray on a premultiplied black RGBA buffer — guarded by a pixel-level test, because a sign flip there would silently invert the whole feature. **Gemini (2.5 Flash Image)** has no mask parameter at all; the mask travels as a second inline image with strict prompt instructions. Whether the model obeys is a quality question, not a correctness one — the client-side compositor enforces the invariant either way. Keys live in the Keychain (`kSecClassGenericPassword`, device-only accessibility). A test dumps `UserDefaults.dictionaryRepresentation()` and asserts the key is not in there. Because only the crop is uploaded, provider output-resolution caps stop mattering: the model sees a ~1024-pixel patch, and a 24-megapixel export keeps its 24 megapixels everywhere the model did not work. ### The auto-detector, and why it is opt-in Version one flagged pixels that were bright and unsaturated — textbook specular highlights. On real museum photos it marked 30–56% of the image (pale skies, a beige dress, the gallery wall) while missing the actual reflections, because a cyan skylight streak is saturated and a soft sheen is below any global brightness bar. Version two used pure local contrast (a morphological white top-hat); paintings are full of bright-next-to-dark, and the overlays looked like a crime scene. The shipped detector is a precision-first hybrid: bright **and** unsaturated **and** locally elevated above its morphological opening, with a minimum-blob filter and the wall-margin band excluded outright. Coverage on the same photos dropped to 0.2–6%, sitting on the actual glare. The cost asymmetry drives it — a missed reflection costs one brush stroke, a false positive costs scrubbing a whole painting's worth of wrong mask. User feedback pushed it further: the mask screen now opens empty with a brush, and auto-detection is a button. ### Brush mechanics The canvas is wrapped in a `UIScrollView` via `UIViewRepresentable` with `panGestureRecognizer.minimumNumberOfTouches = 2`: one finger brushes, two fingers pan, pinch zooms with native physics. Brush radius divides by the zoom scale, so at 4× you paint 4× finer in image pixels. The coordinate math needed no changes — gesture locations inside a zoomed scroll view arrive in the content's unzoomed space, exactly what the display mapper already expects. In-flight strokes render as vector paths during the drag and hand off to the async raster on completion, so there is no finger-up latency. Late async results are handled with a monotonic generation counter, bumped on teardown, captured before every await, checked before every write-back — verified by a test with a gated mock provider that releases its result after the user has exited the screen. ## Testing and tooling * **Swift Testing** for the UI-free geometry and pipeline code, with synthetic fixtures: a factory draws known quads (axis-aligned, rotated, keystoned) into a `CGBitmapContext`, so ground truth is exact by construction and nothing is bundled. * **Pixel-sampling assertions** rather than golden files: after correction, the center must be painting-dark, all four corner regions must be dark, and with a margin the border band must be background-light. Behavior, not bytes. * **Nearest-neighbor corner matching** with ~2.5% tolerances for Vision tests, since Vision is neither pixel-exact nor guaranteed in its corner ordering. * **XCUITest** against the real `PhotosPicker` and the real permission flow, end to end through save. * **`URLProtocol` stubs** for every provider test — multipart field, header, and JSON body assertions with canned responses, no live API in the suite. Coverage sits at 92% across the app target and 100% on the pipeline; the remainder is defensive error branches. The Xcode project is generated by XcodeGen from a ~60-line `project.yml`, the `.xcodeproj` never enters git, and there are zero third-party dependencies. ## Two gotchas worth repeating **Gemini free-tier keys fail misleadingly.** The key validates fine (listing models is free), then every image-generation call returns HTTP 429 permanently. That is not rate limiting — the free tier's quota for the image model is effectively zero, and Google reports "no quota" as `RESOURCE_EXHAUSTED`. Enable billing on the key's project. The app now surfaces the provider's own error body instead of mapping 429 to an optimistic "try again shortly." **`CGImage.cropping(to:)` uses a top-left origin** while the rest of the app lives in Core Image's lower-left space. That flip lives in exactly one wrapper function with a loud comment. ## Availability | ---|--- **App**| PictureFramerApp, App Store ID 6790701502 **Price**| Free **Category**| Graphics & Design **Requirements**| iOS 17.0 / iPadOS 17.0 or later **Size**| 5.9 MB **Language**| English **Age rating**| 4+ **Privacy**| Data Not Collected The straightening pipeline runs entirely on device and makes no network requests. Reflection removal is optional, bring-your-own-key (OpenAI or Google Gemini), and fires only when you tap Remove — a typical removal costs a few cents, billed by your provider. There is no backend, no account, and no subscription. * **App Store:** https://apps.apple.com/ch/app/pictureframerapp/id6790701502 * **Website:** https://pictureframer.corti.com * **Build write-up (geometry pipeline):** https://corti.com/pictureframer-straightening-museum-photos-with-vision-core-image-and-one-carefully-placed-y-flip/ * **Build write-up (reflection removal):** https://corti.com/teaching-my-iphone-app-to-see-through-museum-glass/

My little iOS app PictureFramer does one thing: you photograph a framed painting in a museum, and it straightens the photo — finds the frame, fixes the perspective, keeps a clean strip of wall around it. Pure on-device geometry, no network, done. Then I looked at my camera roll. Half my museum photos had something the geometry pipeline can't fix: **glass**. Skylight streaks smeared across a Hammershøi, a green emergency-exit sign glowing in the varnish of a dark oil painting, spotlights blooming on protective glazing. The perspective was perfect; the painting was still ruined. This is the story of adding AI-powered reflection removal to the app — and the three design decisions, two dead ends, and one billing surprise along the way. ## The one rule: never touch pixels outside the mask Reflection removal means _inventing_ pixels — reconstructing what the artwork looks like under the glare. That's a job for a generative model, and generative models have a well-earned reputation for "improving" things you didn't ask them to touch. For photos of artwork, that's disqualifying. Nobody wants an AI subtly repainting brushwork a painter put there 130 years ago. So the architecture starts from a hard invariant, enforced on the client, not trusted to the model: > **Every pixel outside the user's mask is bit-identical to the original. Not "visually identical" — bit-identical.** The flow that guarantees it: 1. The user marks the glare with a finger (more on that UX below), producing a grayscale mask — white means "repaint". 2. The app crops a padded bounding box around the mask, resizes it to the provider's upload size, and sends _only that crop_ plus the mask to the AI. 3. The returned patch is resized back and composited into the full-resolution image **through a Core Graphics clip mask** — `CGContext.clip(to:mask:)` with the original drawn first. Where the mask is black, the framebuffer simply keeps the original bytes. No Core Image, no color-managed round trip, no drift. 4. The mask edge gets a Gaussian feather so the seam blends — but the feather is multiplied by the binary mask first, so softness only ever grows _inward_. It cannot leak a single pixel past the boundary. There's a unit test that iterates all 32,000 pixels of a fixture and asserts exact equality outside the mask. It's the most important test in the feature. A pleasant side effect of sending only the crop: provider output-resolution caps stop mattering. The model sees a 1024-pixel patch; your 24-megapixel export keeps its 24 megapixels everywhere the AI didn't work. ## Two providers, one protocol, one asymmetry I wanted users to bring their own API key rather than run my images through a server of mine (the app has no backend, and I intend to keep it that way). Two providers made the cut, behind a four-line protocol: protocol InpaintingProvider: Sendable { func uploadSize(for cropSize: CGSize) -> CGSize func inpaint(image: CGImage, mask: CGImage, apiKey: String) async throws -> CGImage } **OpenAI (`gpt-image-1`)** has a real inpainting API: `images/edits` takes an image plus a mask where _transparent_ pixels mark the repaint region. My masks are white-means-repaint grayscale, so there's a small conversion — alpha = 255 − gray, on a premultiplied black RGBA buffer. A pixel-level unit test guards the inversion, because a sign flip there would silently invert the entire feature. **Gemini (2.5 Flash Image)** has no mask parameter at all. The mask travels as a _second inline image_ with strict prompt instructions ("repaint only areas that are white in the mask"). Does the model always obey? It doesn't have to — the client-side compositor enforces the invariant regardless of what comes back. That's the quiet payoff of step 3 above: prompt adherence became a quality concern instead of a correctness concern. API keys live in the Keychain (`kSecClassGenericPassword`, device-only accessibility), never in UserDefaults — and there's a test that dumps `UserDefaults.dictionaryRepresentation()` and asserts the key isn't in there. The detector that marked entire paintings I wanted the app to propose a glare mask automatically. Version one was the obvious heuristic: a pixel is glare if it's **bright and unsaturated** (specular highlights wash out color). It worked beautifully on my synthetic test fixtures. Then I pointed it at real museum photos and it marked **30–56% of the image**. Pale painted skies, a beige dress, the white gallery wall — all bright, all unsaturated, all flagged. Meanwhile it _missed_ the actual reflections, because a cyan skylight streak is colored (saturation above the cutoff) and a soft sheen is dimmer than the global brightness bar. Attempt two: pure local contrast. Glare is additive light, so mark anything brighter than its local surroundings — a morphological white top-hat (luminance minus its opening). Elegant theory. On real paintings, catastrophically wrong in a different way: a painting is _full_ of bright-things-next-to-dark-things. Every pale area within the window of a dark figure lit up. I rendered the detector output as red overlays on my test photos, and the result looked like a crime scene. That debugging loop — a tiny standalone Swift probe that compiles the detector sources with `swiftc`, runs them over real HEIC photos, and writes red-tinted overlay PNGs — turned out to be the most valuable tooling of the project. Thresholds you tune blind are lies; thresholds you tune against overlays converge in two iterations. The version that shipped is a **precision-first hybrid** : a pixel is proposed only if it's bright _and_ unsaturated _and_ locally elevated above its morphological opening, with a minimum-blob filter to kill speckle and the wall-margin band excluded entirely (the margin is real wall — bright by nature, never glare worth fixing). Coverage on the same photos dropped to 0.2–6%, sitting right on the actual glare. The philosophy behind "precision-first" is a cost asymmetry: a **missed** reflection costs the user one brush stroke; a **false positive** costs scrubbing an entire painting's worth of wrong mask. Optimize accordingly. In the end, user feedback pushed this to its logical conclusion — auto-detection is now opt-in behind an _Auto-detect_ button, and the screen opens with an empty mask and a brush. ## The brush that had to earn its keep The mask editor went through three rounds of on-device feedback, each a small lesson in touch UX: **Round one: strokes only appeared on finger-up.** The committed mask is rasterized asynchronously (detector proposal + strokes → grayscale bitmap → red tint), and that pipeline only ran when a stroke ended. The fix is a classic drawing-app pattern: render the in-flight stroke as a _vector_ path live during the drag, and keep the last committed stroke's vector on screen until the async raster catches up, then hand off. No flicker, no lag. **Round two: no zoom.** Precise glare often needs strokes a few points wide, and fingers are fat. Rather than fight SwiftUI's gesture system (which can't cleanly distinguish one-finger from two-finger drags), I wrapped the canvas in a `UIScrollView` via `UIViewRepresentable` and set one property: scrollView.panGestureRecognizer.minimumNumberOfTouches = 2 One finger brushes, two fingers pan, pinch zooms with native physics. The brush radius divides by the zoom scale, so zoomed in 4× you're painting 4× finer in image pixels — precision for free. The coordinate math needed _no changes_ : gesture locations inside a zoomed `UIScrollView` arrive in the content's own unzoomed coordinate space, which is exactly what the existing display-to-canonical mapper expects. **Round three: a Clear button.** When an auto-proposal isn't wanted, erasing it blob by blob is punishment. One button, one `mask.clear()`. ## Gotchas worth writing down **Gemini free-tier keys fail in the most misleading way possible.** The key validates fine in Settings (listing models is free) and then _every_ image-generation call returns HTTP 429 — forever. Not rate limiting: the free tier's quota for the image model is effectively zero, and Google reports "no quota" as `RESOURCE_EXHAUSTED`. My app dutifully mapped 429 to "The provider is rate-limiting — try again shortly," which sent me retrying for hours. The fix is billing on the key's project — and an app change to surface the provider's own error body ("You exceeded your current quota, please check your plan and billing details") instead of my optimistic guess. If your error mapping throws away the response body, you're throwing away the diagnosis. **`CGImage.cropping(to:)` uses a top-left origin.** The entire app lives in Core Image's lower-left coordinate space; this one API doesn't. The flip lives in exactly one wrapper function with a loud comment, because coordinate bugs metastasize. **Async results can outlive the screen that requested them.** Fire off a 10-second inpainting call, and the user might cancel, change the crop, or export before it lands. A late result must not resurrect state the user tore down. A monotonic generation counter — bumped on every teardown, captured before every await, checked before every write-back — closed that hole, verified by a test with a gated mock provider that deliberately releases its result _after_ the user has exited. ## Testing without a network Every provider test runs against a `URLProtocol` stub — request assertions (multipart fields, headers, JSON body shape) and canned responses, no live API anywhere in the suite. The one thing stubs can't verify is the real contract: the actual first live call caught a modality quirk in the Gemini request that no fixture would ever have found. Stub everything, then smoke-test each provider once with a real key before shipping. Both lessons are old; both apparently need relearning every project. ## The result The green exit sign is gone from the Hammershøi. The brushwork around it is untouched — provably, byte-for-byte. And the whole feature stays true to the app's original privacy posture: no backend, no accounts, and the only network calls are the ones you explicitly trigger, to the provider you chose, with your own key. _PictureFramer is iPhone-only, iOS 17+. The straightening pipeline runs entirely on-device; reflection removal is optional and bring-your-own-key (OpenAI or Google Gemini)._

I take a lot of photos of paintings in museums. They all have the same problem: you can rarely stand dead center in front of the artwork, so every photo is a little rotated and keystoned, with the fra...

I take a lot of photos of paintings in museums. They all have the same problem: you can rarely stand dead center in front of the artwork, so every photo is a little rotated and keystoned, with the frame converging toward one side. Cropping doesn't fix perspective, and generic document scanners crop _to_ the edge — I wanted the frame _and_ a clean strip of the wall around it, like a catalog photograph. So I built PictureFramer, an iPhone app that imports a photo of a framed painting, finds the outer edge of the artwork automatically, corrects rotation and keystone in one transform, keeps a configurable margin of real wall pixels around the frame, and saves the result at full resolution. This post covers the architecture decisions that made it pleasant to build — and the platform surprises that didn't. App website: pictureframer.corti.com. The app is currently in preview on TestFlight. ## The one decision that mattered: a canonical coordinate space Image pipelines on iOS juggle at least three coordinate systems: Vision returns normalized coordinates with a lower-left origin, Core Image works in pixels with a lower-left origin, and UIKit/SwiftUI draw with a top-left origin. Most of the classic bugs in this kind of app are silent flips and scale confusions between these spaces. The fix was declaring one canonical space up front: **full-resolution source-image pixels, lower-left origin** — deliberately identical to Core Image's space. Everything speaks it: * Vision → canonical is a pure scale. No flip, because both are lower-left. One tiny function, `VisionQuadConversion`, is the only code in the app allowed to interpret Vision's normalized output. * Canonical → `CIPerspectiveCorrection` is the identity. The detected corners pass straight into the filter as `CIVector`s. * The **only y-flip in the entire app** lives in one type, `DisplayMapper`, at the SwiftUI boundary. It maps canonical pixels to the aspect-fitted image's display points and back. Every gesture — corner drags, preview panning — converts through it. A `Quad` struct (four `CGPoint`s in canonical space) is the single currency of the pipeline. Detection may run on a downscaled copy for speed, but the detector converts to full-res pixels _before returning_ , so a "which scale is this quad in?" bug can't exist by construction. ## Margin means real wall, not padding The feature I cared most about: after straightening, keep N pixels of background around the frame — the actual wall from the photo, not synthetic border fill. That means the margin has to be applied _before_ perspective correction, in source space. Expanding a tilted quad isn't just insetting a rectangle negatively: each edge gets offset outward along its outward normal (computed against the centroid, so winding order doesn't matter), and adjacent offset edge lines are re-intersected to find the new corners. The expanded quad then samples real background pixels through the same homography as the painting itself. Two edge cases bit during testing: * **Oversized margins** (more margin than available wall) clamp per-corner to the image bounds, gracefully degrading to "use everything up to the photo's edge." * **Negative margins** (shrinking) can collapse the quad past zero and flip its winding — and a winding-flipped quad still passes a naive convexity test, because all the cross products just change sign together. A shoelace-formula signed-area check that compares winding before and after expansion catches it. A unit test found this one before any user could. ## Detection: Vision with a permissive fallback `VNDetectRectanglesRequest` does the heavy lifting, tuned for "a large framed rectangle filling much of the photo" (high minimum size, wide aspect-ratio range). When that finds nothing — small artworks, extreme panoramas, low-contrast frames — a second pass runs with permissive thresholds. The best observation wins by confidence, with area as the tie-breaker so the outer frame edge beats an inner mat edge. Against my eight real museum test photos (Hammershøi, mostly, photographed handheld at whatever angle the crowd allowed), the default configuration detected 8/8 — including an unframed canvas, where the stretcher edge was enough. Detection failure in the app falls back to a centered draggable quad, so the user is never stuck. ## Testing an image pipeline without golden files All the geometry and pipeline code is UI-free and unit-tested with Swift Testing. The interesting part is testing _image_ behavior headlessly: * **Synthetic fixtures** : a test factory draws known quads (axis-aligned, rotated, keystoned) with a lighter "frame" stroke into a `CGBitmapContext`. Deterministic, no bundled assets, and the ground truth is exact by construction. * **Pixel-sampling assertions** : after correction, sample the output — center must be painting-dark, all four corner regions must be dark (proof it straightened), and with a margin, the border band must be background-light (proof the margin is real wall, not padding). Behavior, not bytes. * **Nearest-neighbor corner matching** with tolerances around 2.5% of the image dimension for Vision tests — Vision is not pixel-exact and its corner ordering isn't guaranteed. One calibration surprise: `CIPerspectiveCorrection` doesn't produce an output sized like the quad's edge lengths. It reconstructs the rectangle's _true_ proportions via the homography — a keystoned quad's output can be 25% taller than its average edge length. The size assertions had to test neighborhoods, not exact values; the behavioral pixel assertions stayed strict. Coverage across the app target sits at 92%, with the pipeline itself at 100%. The remaining gap is defensive error branches. ## An end-to-end test that fights the photo picker The XCUITest drives the _real_ `PhotosPicker` and the _real_ permission flow: pick the newest photo, check the auto-detected quad, drag a corner, pan the preview, save, verify the success screen. Things I now know about automating the iOS 26 picker: * Grid cells are exposed as images with identifier `PXGGridLayout-Info`, newest first. * A first-run onboarding banner can cover the grid; close it. * Cells report non-hittable while thumbnails stream in — coordinate-tap them. * **iOS 26 auto-grants add-only photo saves.** With the permission not-determined, `PHPhotoLibrary.requestAuthorization(for: .addOnly)` returns `.authorized` with no prompt at all. The denial prompt only exists after an explicit `simctl privacy revoke` — it's a new card-style dialog that springboard accessibility queries can't see, and answering it mid-request can get your app killed for the TCC change. The denial-path test ended up two-phase: deny once, relaunch clean, verify the error UI. ## Tooling notes The Xcode project is generated by XcodeGen — `project.yml` is 60 readable lines, the `.xcodeproj` never touches git, and merge conflicts in project files are a solved problem. Distribution had one more surprise. My iPhone is MDM-enrolled and the policy blocks Developer Mode, so cable deploys were out — TestFlight was the workaround (distribution builds don't need Developer Mode). But `xcodebuild archive` with automatic signing wants a _development_ provisioning profile, which requires a registered device — which I couldn't register. The escape hatch: archive unsigned (`CODE_SIGNING_ALLOWED=NO`), then let `-exportArchive -allowProvisioningUpdates` sign with the App Store distribution profile, which needs no devices at all. Transporter took the IPA on the second try — the first bounced because even an iPhone-only app must declare all four iPad orientations (`UISupportedInterfaceOrientations~ipad`) for iPad-compatibility multitasking. ## The stack, summarized * **SwiftUI +`@Observable`** view model; the UI is a thin shell over a tested pipeline * **Vision** (`VNDetectRectanglesRequest`) for detection, two-pass * **Core Image** (`CIPerspectiveCorrection`, one shared `CIContext`) for correction * **PhotosUI / Photos** for import (no permission needed — the picker is out-of-process) and add-only export * **Swift Testing** for units, **XCUITest** for the end-to-end flow, synthetic fixtures throughout * **XcodeGen** for the project, zero third-party dependencies Everything runs on-device; the app makes no network requests. If you photograph paintings and share my crooked-photo affliction: PictureFramer is on its way to the App Store.