The board adopted Resolution of Necessity No. 2026-385 to authorize eminent domain for the I‑80/San Pablo Dam Road interchange improvement (phase 2) after a public hearing; the property owner's representative urged a 30‑day deferral, saying the current design would eliminate the site's sole ingress/egress.

A recall of 265,512 Ram ProMaster models for power steering that may fail due to wiring harness water ingress.

Purifierオニキスメダル到達です。マジで感無量。2015年にブロンズメダルを貰ってから11年ですよ。11年。僕自身が構築よりで破壊が少ない方らしいとはい…

Purifierオニキスメダル到達です。マジで感無量。2015年にブロンズメダルを貰ってから11年ですよ。11年。僕自身が構築よりで破壊が少ない方らしいとはい…

Inbound webhooks are deceptively hard to handle at scale. Providers such as Stripe, GitHub, and Shopify don’t know about your database connection pool, your autoscaling lag, or a brief microservice outage. They send events when things happen, and a busy period for one of your customers can turn into a burst of requests against your origin. If the origin sits directly behind a load balancer, one of those bursts can exhaust database connections and memory and cause timeouts. Some senders then retry, which adds more load while you are already struggling. One way to protect the origin is an **ingress buffer** at the edge. With **Cloudflare Workers** , **Workers KV** , and **Durable Objects** , you can verify signatures, deduplicate, rate-limit, and durably queue incoming webhooks before a single request reaches your infrastructure. Cloudflare runs Workers in more than 330 cities and says that puts code within roughly 50 ms of about 95% of the world’s Internet-connected population. One caveat up front: only the entrypoint Worker is “everywhere”. Each tenant’s Durable Object is a single globally unique instance that coordinates all requests sent to it. The buffer is therefore distributed across tenants, not replicated for any one tenant. ## Why Webhook Ingestion Fails Each of these failure modes shapes the design below. 1. **Retry behavior differs by provider.** Stripe retries failed live-mode deliveries with exponential backoff for up to three days and disables endpoints that keep failing. Shopify retries 8 times over the following 4 hours. GitHub does not automatically redeliver failed deliveries at all. A short outage can therefore produce a burst of retries from some senders and permanently lost events from others. You can’t assume one retry contract. 2. **Senders give up quickly.** Shopify allows a one-second connection timeout and a five-second timeout for the whole request. If your origin does real work before responding, slow responses are treated as failures and trigger the retries you were trying to avoid. 3. **Duplicates are normal.** Providers deliver at least once. Shopify says its app “might receive the same webhook more than once”, for example after a network timeout or a retry, and recommends deduplicating on the `X-Shopify-Webhook-Id` header. Stripe events carry a stable event ID that stays the same across retries. 4. **One tenant can starve the rest.** A misconfigured integration that fires thousands of events can consume shared processing capacity unless you isolate and limit it per tenant. Acknowledging quickly after a durable write handles the first two, deduplication handles the third, and a per-tenant limiter handles the fourth. ## Architecture [Webhook sender] │ ▼ [Worker: validate tenant, verify HMAC] │ │ │ └──► [Workers KV: cached tenant config, secrets, limits] ▼ [Durable Object (one per tenant, SQLite-backed)] ├─ dedupe by event ID ├─ per-tenant rate limit ├─ durable event queue └─ alarm → batched delivery to origin (or to Cloudflare Queues) * **Worker (entrypoint):** Parses the path, loads tenant config, verifies the signature, and forwards verified events to the tenant’s Durable Object. Anycast routing and Cloudflare’s scheduling typically place it close to the sender. * **Workers KV:** A read-optimized, eventually consistent cache for data that changes rarely, such as tenant config and rate limits. Cloudflare cites sub-5 ms hot read latency. Writes can take 60 seconds or more to be visible in other locations. * **Durable Objects:** Provide per-tenant coordination with a private, transactional SQLite database. Because each object processes one tenant’s events, dedupe checks and rate-limit counters don’t need distributed locks. * **Alarms:** Each object can schedule a single alarm, which lets it drain its buffer in batches without waiting for another incoming request. ## Step 1: The Entrypoint Worker The Worker reads the raw bytes, verifies the HMAC over exactly those bytes, and only then touches the Durable Object. Several details differ from naive implementations: * **Fail closed.** If there is no secret for the tenant, the request is rejected. A shared fallback secret would let anyone who knows it forge events for every tenant. * **Constant-time comparison.** Workers provide `crypto.subtle.timingSafeEqual`. Cloudflare’s documentation notes it is not constant-time with respect to length, which is harmless here because an HMAC-SHA256 digest is always 32 bytes. * **Meaningful dedupe keys.** If the sender provides no event ID, the code falls back to a hash of the body. A random UUID would defeat deduplication, because every retry would look new. * **Minimal forwarding.** Only what the origin needs is passed on, not every header. Forwarding all headers would persist signatures and other sensitive values. // src/index.ts import type { TenantWebhookBuffer } from "./tenant-webhook-buffer"; export interface Env { TENANT_DO: DurableObjectNamespace<TenantWebhookBuffer>; WEBHOOK_KV: KVNamespace; ORIGIN_BASE_URL: string; // e.g. https://api.example.com ORIGIN_TOKEN: string; // set with `wrangler secret put ORIGIN_TOKEN` } interface TenantConfig { active: boolean; rateLimit: number; // max accepted events per 10-second window secrets: string[]; // current + previous secret during rotation } const TENANT_ID = /^[A-Za-z0-9_-]{1,64}$/; const encoder = new TextEncoder(); export default { async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> { if (request.method !== "POST") { return new Response("Method Not Allowed", { status: 405, headers: { Allow: "POST" } }); } // Expected path: /webhook/<tenantId> const [, prefix, tenantId] = new URL(request.url).pathname.split("/"); if (prefix !== "webhook" || !tenantId || !TENANT_ID.test(tenantId)) { return new Response("Not Found", { status: 404 }); } const config = await getTenantConfig(env, ctx, tenantId); if (!config?.active) { return new Response("Unknown tenant", { status: 404 }); } // Verify against the exact bytes the sender signed. const bodyBytes = new Uint8Array(await request.arrayBuffer()); const signature = request.headers.get("X-Signature") ?? ""; let valid = false; for (const secret of config.secrets) { if (await verifyHmacHex(bodyBytes, signature, secret)) valid = true; } if (!valid) { return new Response("Invalid signature", { status: 401 }); } const eventId = request.headers.get("X-Event-ID") ?? (await sha256Hex(bodyBytes)); try { const stub = env.TENANT_DO.getByName(tenantId); const result = await stub.ingestWebhook( { eventId, receivedAt: Date.now(), contentType: request.headers.get("Content-Type") ?? "application/octet-stream", rawBody: new TextDecoder().decode(bodyBytes), // assumes UTF-8 payloads (JSON) }, config.rateLimit, ); return Response.json(result.body, { status: result.status, headers: result.retryAfter ? { "Retry-After": String(result.retryAfter) } : undefined, }); } catch { // The event was NOT durably stored, so ask the sender to retry. return new Response("Temporarily unavailable", { status: 503, headers: { "Retry-After": "30" } }); } }, }; async function verifyHmacHex(body: Uint8Array, headerSig: string, secret: string): Promise<boolean> { const provided = hexToBytes(headerSig); if (!provided) return false; const key = await crypto.subtle.importKey( "raw", encoder.encode(secret), { name: "HMAC", hash: "SHA-256" }, false, ["sign"], ); const expected = new Uint8Array(await crypto.subtle.sign("HMAC", key, body)); return provided.length === expected.length && crypto.subtle.timingSafeEqual(provided, expected); } function hexToBytes(hex: string): Uint8Array | null { if (hex.length % 2 !== 0 || !/^[0-9a-fA-F]*$/.test(hex)) return null; const out = new Uint8Array(hex.length / 2); for (let i = 0; i < out.length; i++) out[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16); return out; } async function sha256Hex(data: Uint8Array): Promise<string> { const digest = await crypto.subtle.digest("SHA-256", data); return Array.from(new Uint8Array(digest), (b) => b.toString(16).padStart(2, "0")).join(""); } This example assumes a generic sender that sends a hex HMAC-SHA256 of the body in `X-Signature`. Real providers differ, so adapt the verification step: * GitHub sends `sha256=<hex>` in `X-Hub-Signature-256`. * Shopify sends a base64-encoded digest in `X-Shopify-Hmac-SHA256`. * Stripe signs a timestamp plus the payload, which also gives you replay protection. For providers with a stable delivery ID, use it as the dedupe key: `X-Shopify-Webhook-Id` for Shopify, `X-GitHub-Delivery` for GitHub, and the event `id` in the body for Stripe. The Worker calls the Durable Object through Workers RPC. Public methods on a class that extends `DurableObject` are callable through a stub. Arguments and return values must be serializable, so this example returns a plain object and builds the `Response` in the Worker. ## Step 2: Per-Tenant State in a Durable Object Each tenant’s Durable Object holds four things in its SQLite database: * processed event IDs, for deduplication * a short rate-limit window * the pending event queue * a small counter for delivery backoff SQLite-backed Durable Objects are Cloudflare’s recommended storage backend for new classes, and the SQL API is synchronous. Writes made without an intervening `await` are committed together, and an output gate holds the response until the writes are confirmed. The 202 therefore means the event has been durably stored, not just accepted into memory. Keeping deduplication and rate limiting in separate tables matters. Pruning old rate-limit rows from the same table that records processed IDs would erase your dedupe history after seconds. // src/tenant-webhook-buffer.ts import { DurableObject } from "cloudflare:workers"; import type { Env } from "./index"; export interface WebhookEvent { eventId: string; receivedAt: number; contentType: string; rawBody: string; } export interface IngestResult { status: 200 | 202 | 429; body: Record<string, unknown>; retryAfter?: number; } const RATE_WINDOW_MS = 10_000; const DEDUPE_RETENTION_MS = 4 * 24 * 60 * 60 * 1000; // longer than Stripe's 3-day retry window const BATCH_SIZE = 50; const FLUSH_DELAY_MS = 2_000; const MAX_BACKOFF_MS = 5 * 60 * 1000; export class TenantWebhookBuffer extends DurableObject<Env> { constructor(ctx: DurableObjectState, env: Env) { super(ctx, env); ctx.blockConcurrencyWhile(async () => { const sql = ctx.storage.sql; sql.exec(`CREATE TABLE IF NOT EXISTS processed_events ( event_id TEXT PRIMARY KEY, received_at INTEGER NOT NULL)`); sql.exec(`CREATE TABLE IF NOT EXISTS rate_window (received_at INTEGER NOT NULL)`); sql.exec(`CREATE INDEX IF NOT EXISTS idx_rate_window ON rate_window (received_at)`); sql.exec(`CREATE TABLE IF NOT EXISTS event_queue ( id INTEGER PRIMARY KEY AUTOINCREMENT, event_id TEXT NOT NULL, payload TEXT NOT NULL, received_at INTEGER NOT NULL)`); sql.exec(`CREATE TABLE IF NOT EXISTS flush_state ( id INTEGER PRIMARY KEY CHECK (id = 1), failures INTEGER NOT NULL DEFAULT 0)`); sql.exec(`INSERT OR IGNORE INTO flush_state (id, failures) VALUES (1, 0)`); }); } async ingestWebhook(event: WebhookEvent, limit = 100): Promise<IngestResult> { const sql = this.ctx.storage.sql; const now = Date.now(); // 1. Deduplicate. Duplicates are acknowledged with 200 and cost almost nothing. const seen = sql.exec("SELECT 1 FROM processed_events WHERE event_id = ?", event.eventId).toArray(); if (seen.length > 0) { return { status: 200, body: { status: "deduplicated", eventId: event.eventId } }; } // 2. Sliding-window rate limit over accepted events. sql.exec("DELETE FROM rate_window WHERE received_at < ?", now - RATE_WINDOW_MS); const { count } = sql.exec<{ count: number }>("SELECT COUNT(*) AS count FROM rate_window").one(); if (count >= limit) { return { status: 429, body: { status: "rate_limited" }, retryAfter: 10 }; } // 3. Record and enqueue atomically. this.ctx.storage.transactionSync(() => { sql.exec("INSERT INTO processed_events (event_id, received_at) VALUES (?, ?)", event.eventId, now); sql.exec("INSERT INTO rate_window (received_at) VALUES (?)", now); sql.exec( "INSERT INTO event_queue (event_id, payload, received_at) VALUES (?, ?, ?)", event.eventId, JSON.stringify(event), now, ); }); // Schedule a flush only if one isn't pending. Calling setAlarm() again would // overwrite the pending alarm and keep pushing it back under steady traffic. if ((await this.ctx.storage.getAlarm()) === null) { await this.ctx.storage.setAlarm(now + FLUSH_DELAY_MS); } return { status: 202, body: { status: "buffered", eventId: event.eventId } }; } async alarm(): Promise<void> { const sql = this.ctx.storage.sql; // Housekeeping: expire old dedupe records. sql.exec("DELETE FROM processed_events WHERE received_at < ?", Date.now() - DEDUPE_RETENTION_MS); const rows = sql .exec<{ id: number; payload: string }>("SELECT id, payload FROM event_queue ORDER BY id LIMIT ?", BATCH_SIZE) .toArray(); if (rows.length === 0) return; let delivered = false; try { const response = await fetch(`${this.env.ORIGIN_BASE_URL}/internal/webhooks/batch`, { method: "POST", headers: { "Content-Type": "application/json", Authorization: `Bearer ${this.env.ORIGIN_TOKEN}`, }, body: JSON.stringify({ events: rows.map((r) => JSON.parse(r.payload)) }), signal: AbortSignal.timeout(10_000), }); delivered = response.ok; } catch { delivered = false; } if (delivered) { sql.exec("DELETE FROM event_queue WHERE id <= ?", rows[rows.length - 1].id); sql.exec("UPDATE flush_state SET failures = 0 WHERE id = 1"); const { n } = sql.exec<{ n: number }>("SELECT COUNT(*) AS n FROM event_queue").one(); if (n > 0) await this.ctx.storage.setAlarm(Date.now() + FLUSH_DELAY_MS); // keep draining } else { const { failures } = sql.exec<{ failures: number }>("SELECT failures FROM flush_state WHERE id = 1").one(); sql.exec("UPDATE flush_state SET failures = ? WHERE id = 1", failures + 1); const delay = Math.min(FLUSH_DELAY_MS * 2 ** failures, MAX_BACKOFF_MS); await this.ctx.storage.setAlarm(Date.now() + delay); } } } A few design points are worth explaining: * **Alarm retries are limited.** Cloudflare retries a throwing `alarm()` handler with exponential backoff starting at 2 seconds, up to 6 retries. After that the alarm is not retried. That is not enough to ride out a long origin outage, so the handler above catches errors and manages its own backoff, capped at five minutes. * **Delivery to the origin is at-least-once.** If the Durable Object is interrupted after the origin accepted a batch but before the rows are deleted, the batch is sent again. Each event carries its `eventId`, so the origin should deduplicate on it. * **A permanently failing batch blocks the head of the queue.** In production, cap per-event attempts and move poison events to a dead-letter table. * **The 429 is a safety valve, not a guarantee.** Providers that retry, such as Stripe and Shopify, will treat a non-2xx as a failure and try again later. Don’t rely on every sender honoring `Retry-After`, and GitHub won’t redeliver automatically. ## Step 3: Workers KV for Tenant Config KV suits data that is read often and changed rarely, such as tenant status, rate limits, and signing secrets. It is eventually consistent. A write is visible immediately in the location that made it, but can take 60 seconds or more to propagate elsewhere, and a missing key can be cached for up to that long too. KV also enforces a minimum TTL of 60 seconds and a maximum of one write per second to the same key. That has practical consequences: * **Use short TTLs.** A 300-second `expirationTtl` is a reasonable default for rules that change occasionally. * **Accept old and new secrets during rotation.** The `secrets` array in the config above does this, because a rotated secret can lag at some edge locations. * **Don’t write on every cache miss.** Concurrent misses for one key would hit the write limit, so the write happens in `ctx.waitUntil` and failures are ignored. * **Don’t cache unknown tenants.** Otherwise anyone can fill your namespace with junk keys by inventing tenant IDs. async function getTenantConfig(env: Env, ctx: ExecutionContext, tenantId: string): Promise<TenantConfig | null> { const key = `config:${tenantId}`; const cached = await env.WEBHOOK_KV.get<TenantConfig>(key, { type: "json", cacheTtl: 60 }); if (cached) return cached; // Cache miss: load from the system of record. Only known tenants are cached. const response = await fetch(`${env.ORIGIN_BASE_URL}/internal/tenants/${tenantId}/webhook-config`, { headers: { Authorization: `Bearer ${env.ORIGIN_TOKEN}` }, signal: AbortSignal.timeout(3_000), }); if (!response.ok) return null; const fresh = (await response.json()) as TenantConfig; ctx.waitUntil( env.WEBHOOK_KV.put(key, JSON.stringify(fresh), { expirationTtl: 300 }).catch(() => {}), ); return fresh; } Treat KV as a cache of data you can reload from your system of record. Never hardcode secrets in source. ## Wrangler Configuration Cloudflare now documents a declarative `exports` field for Durable Object class lifecycle in new Workers. Older projects use the `migrations` array with `new_sqlite_classes`, and a Worker can use only one of the two. Cloudflare recommends the SQLite backend for all new namespaces. // wrangler.jsonc { "name": "webhook-ingress", "main": "src/index.ts", "compatibility_date": "2026-10-01", "vars": { "ORIGIN_BASE_URL": "https://api.example.com" }, "kv_namespaces": [{ "binding": "WEBHOOK_KV", "id": "<KV_NAMESPACE_ID>" }], "durable_objects": { "bindings": [{ "name": "TENANT_DO", "class_name": "TenantWebhookBuffer" }] }, "exports": { "TenantWebhookBuffer": { "type": "durable-object", "storage": "sqlite" } } } If you are on the legacy flow, replace `exports` with `"migrations": [{ "tag": "v1", "new_sqlite_classes": ["TenantWebhookBuffer"] }]`. ## Failure Modes and Mitigations Component | Consistency model | Primary failure mode | Mitigation ---|---|---|--- Worker | Stateless | Uncaught exception or timeout | Wrap the Durable Object call in `try/catch` and return a deliberate 503 so retrying senders try again Workers KV | Eventually consistent | Stale config or secrets for 60 seconds or more | Short TTLs, accept old and new secrets during rotation, don’t cache unknown tenants Durable Object | Strongly consistent per object | Per-object limits: 10 GB storage on Workers Paid (1 GB on Free), soft limit of about 1,000 requests per second | Alert on backlog depth, split very large tenants across several objects, and shard dedupe by event ID Alarm | At-least-once | Handler keeps failing after the built-in retries | Custom backoff with a cap; origin deduplicates by `eventId` Origin | Yours | Outage or slow responses | The buffer absorbs it; drain in controlled batches ### Backpressure and Origin Outages If the origin goes down, each tenant’s Durable Object keeps accepting and storing events up to its storage limit, and the sender sees fast 202 responses. When the origin recovers, the alarm drains the backlog in batches of 50 with a pause between them, which avoids a thundering herd against your database. Buffered events are only safe while there is space. When a SQLite-backed Durable Object hits its storage cap, writes fail. In this design that surfaces as a 503, and providers that retry will redeliver later. Providers that don’t retry, GitHub among them, will have lost the event. Monitor queue depth and age so you can act well before the cap. ## Alternative: Cloudflare Queues If you prefer to hand batches to a managed queue, Cloudflare Queues offers at-least-once delivery, consumer batches of up to 100 messages, and dead-letter queues. A single message is limited to 128 KB, so large webhook bodies need to go to object storage with only a reference queued. A Durable Object buffer, as built above, gives you per-tenant ordering, deduplication, and rate limiting in one place, and Queues can still sit behind it as the delivery mechanism. ## Production Checklist 1. **Fail closed on authentication.** No fallback secrets, constant-time comparison, and provider-specific verification. Check timestamps where the provider signs them. 2. **Deduplicate on the provider’s ID and retain it long enough.** Keep dedupe records for longer than the longest provider retry window. Stripe’s is three days, hence four above. 3. **Make the origin idempotent.** Delivery from the buffer is at-least-once, so the origin must tolerate repeats. 4. **Set limits deliberately.** Choose per-tenant rate limits from real traffic, and plan for tenants that approach a single object’s throughput ceiling. 5. **Add a dead-letter path** for events the origin permanently rejects, so they don’t block a tenant’s queue. 6. **Alert on backlog depth and oldest-event age** per tenant. 7. **Test the failure paths** : origin down, KV stale, duplicate floods, and a rotated secret. ## Further Reading * Durable Objects alarms * Durable Objects limits * Durable Object class exports * Workers RPC * Workers KV: writing key-value pairs * Protecting against timing attacks with timingSafeEqual * Shopify: verifying webhook deliveries * GitHub: handling failed webhook deliveries * Hookdeck: guide to Stripe webhooks _Originally published at https://instawebhook.com/blog/routing-webhooks-through-cloudflare-workers-kv-and-durable-objects-building-a-gl_

Alt: Lien Twining Novel Ingress