Restore file SQL.GZ vào LIBSQL trên AlmaLinux Nếu server của bạn đang chạy AlmaLinux, cách ổn định nhất là không cần giải nén file .sql.gz ra ổ đĩa, mà stream trực tiếp qua gzip -dc hoặc zcat vào Turso CLI. 1. Kiểm tra Turso CLI đã cài chưa turso --version Nếu báo command not found thì cài Turso CLI…

Restore file SQL.GZ vào LIBSQL trên AlmaLinux Nếu server của bạn đang chạy AlmaLinux, cách ổn định nhất là không cần giải nén file .sql.gz ra ổ đĩa, mà stream trực tiếp qua gzip -dc hoặc zcat vào Turso CLI. 1. Kiểm tra Turso CLI đã cài chưa turso --version Nếu báo command not found thì cài Turso CLI…

Restore file SQL.GZ vào LIBSQL trên AlmaLinux Nếu server của bạn đang chạy AlmaLinux, cách ổn định nhất là không cần giải nén file .sql.gz ra ổ đĩa, mà stream trực tiếp qua gzip -dc hoặc zcat vào Turso CLI. 1. Kiểm tra Turso CLI đã cài chưa turso --version Nếu báo command not found thì cài Turso CLI…

Restore file SQL.GZ vào LIBSQL trên AlmaLinux Nếu server của bạn đang chạy AlmaLinux, cách ổn định nhất là không cần giải nén file .sql.gz ra ổ đĩa, mà stream trực tiếp qua gzip -dc hoặc zcat vào Turso CLI. 1. Kiểm tra Turso CLI đã cài chưa turso --version Nếu báo command not found thì cài Turso CLI…

The full pipeline for generating X-vs-Y comparison pages across 1,500+ HuggingFace models — pairing logic, Claude Haiku generation, and pair_slug caching in Turso.

My AI tools directory (aiappdex.com) launched with individual model pages — one page per HuggingFace model, with download counts, tags, and Claude-generated pros and cons. That's a minimum viable directory. But directory users don't only search for a model by name; they search for two models side by side. "Llama 3 vs Mistral 7B for a local inference project." "Which text-classification checkpoint fits my fine-tuning budget." Comparison pages serve a different intent than detail pages, and I built a pipeline to generate them at scale. This is how the pipeline works, what its failure modes are, and what I'd change if I were starting over. ## Why comparison pages at all The argument for comparison pages is search intent. A user who types "[model A] vs [model B]" is further along in an evaluation than one who types "[model A]." They already know which models they're considering; they want a structured side-by-side. That's a more specific intent signal. The argument against auto-generating them is quality. If I ask Claude to compare two models it knows nothing concrete about and produce a paragraph of accurate-sounding generalities, I'm not adding value — I'm producing plausible boilerplate. The prompting had to produce content specific enough to be useful, or at least honest enough not to actively mislead. The experiment I'm running has a limited monthly budget (~$25), so the comparison generation had to be economical. I'm using Claude Haiku for this step, for the same reasons I use it across all three directory ETLs: fast, cheap, and reliably produces well-formed JSON when the prompt is constrained. ## The pairing problem With ~1,500 models in the database, the number of possible pairs is roughly 1.1 million. Generating and caching a million comparisons isn't feasible. I needed to reduce that to something tractable and coherent. **First instinct: pair by download count.** The top-N models globally give you the highest-profile pairs — Llama vs Mistral, bert-base-uncased vs roberta-base. This produces a handful of defensible comparisons for popular queries, but it ignores the long tail almost entirely. My directory is specifically built for the long tail where the big-name aggregators don't focus. **What I settled on: group by pipeline tag, then pair the top 4 within each group.** HuggingFace's pipeline tags roughly correspond to task categories: `text-generation`, `text-classification`, `image-to-image`, `token-classification`, and so on. A user evaluating a text-classification model cares about comparisons to other text-classification models, not about image generators from the same organization. const byPipe = new Map<string, typeof models>(); for (const m of models) { if (!m.pipeline_tag) continue; const arr = byPipe.get(m.pipeline_tag) ?? []; arr.push(m); byPipe.set(m.pipeline_tag, arr); } const pairs: Array<[Model, Model]> = []; for (const [, list] of byPipe) { const sorted = [...list].sort((a, b) => b.downloads - a.downloads); const take = sorted.slice(0, Math.min(4, sorted.length)); for (let i = 0; i < take.length; i++) { for (let j = i + 1; j < take.length; j++) { pairs.push([take[i]!, take[j]!]); } } } For a pipeline tag with 4 top models, that produces 6 pairs (4 choose 2). Across roughly 30 active pipeline tags in the database, that's around 180 pairs total — manageable, and where each pair is at least plausibly relevant to the same user. I cap total pairs per run at 50 using a `COMPARE_LIMIT` env variable. Most runs skip most of those pairs because they already exist in the database. ## Claude Haiku generation with a fallback template The generation step asks Haiku for a structured JSON comparison. My general approach to Claude JSON extraction carries directly into this pipeline: constrain the system prompt to a single JSON schema, match-and-parse the response, fall back to a template on failure. The system prompt: You compare two AI models for a technical directory. Given two model names and their metadata, produce a structured comparison. Output ONLY a JSON object: { "summary": "2-sentence overview", "differences": ["3-5 bullet points on key differences"], "similarities": ["2-3 bullet points on what they share"], "recommendation": "1-2 sentence guidance on which to pick in which situation" } Be concrete and developer-focused. No markdown, no prose outside the JSON. The user prompt passes both models' names, authors, pipeline tags, and their existing summary strings from the database (capped at 400 characters each). That gives Haiku concrete inputs without requiring it to hallucinate architecture details. The fallback template is not good content, but it's not wrong content either: const fb: CompareData = { summary: `${a.name} and ${b.name} are both ${a.pipeline_tag} models. See each entry for specifics.`, differences: ["See individual model pages for architecture and use cases."], similarities: ["Both are open-source models on HuggingFace."], recommendation: "Pick based on your compute budget and specific task requirements.", }; This template is honest about what it doesn't know. The page renders, the structure is valid, and if I run the pipeline again with a working API key, the caching layer (described next) handles replacement cleanly. ## libSQL caching with pair_slug UNIQUE key The most important design decision in this pipeline is the dedup layer. Every pair gets a deterministic slug: const pairSlug = [a.slug, b.slug].sort().join("--vs--"); Sorting before joining ensures `model-a--vs--model-b` and `model-b--vs--model-a` produce the same key regardless of which model appears first in the loop iteration. This slug is a `UNIQUE` column in the `model_compare` table. Before calling Claude for any pair, the pipeline checks: const existing = await db.execute({ sql: `SELECT 1 FROM model_compare WHERE pair_slug = ?`, args: [pairSlug] }); if (existing.rows.length > 0) { skipped++; continue; } Existing rows are never regenerated. This makes the daily cron idempotent with no extra bookkeeping: the table itself is the state. A run that processes 50 candidates might generate only 3-5 new comparisons if most pairs are already cached. Turso (libSQL) is my database layer for all three sites. The pair_slug UNIQUE constraint relies on slug stability — the slug collision I fixed last week is relevant here because a pair_slug composed of two model slugs inherits any instability from either component slug. If model A's slug changes between runs, the old pair row becomes an orphan. ## Exporting to JSON for Astro SSG After all pairs are processed, the pipeline dumps the full `model_compare` table to a JSON file: const all = await db.execute(`SELECT * FROM model_compare ORDER BY slug_a, slug_b`); const entries = all.rows.map((r) => ({ slug_a: String(r.slug_a), slug_b: String(r.slug_b), pair_slug: String(r.pair_slug), summary: r.summary ? String(r.summary) : "", differences: r.differences ? JSON.parse(String(r.differences)) : [], similarities: r.similarities ? JSON.parse(String(r.similarities)) : [], recommendation: r.recommendation ? String(r.recommendation) : "", })); await writeFile("./src/data/compare.json", JSON.stringify(entries, null, 2)); Astro generates a static page for each `pair_slug` at `/compare/[pair_slug]`. I'm using SSG because static pages cost nothing to serve at current traffic levels and the comparison content doesn't change per-visitor. The `differences` and `similarities` arrays are stored as JSON strings in libSQL — a JSON blob approach rather than normalized rows. For a few hundred comparisons, this is fast enough, and it keeps the schema simple. ## What worked **The fallback template makes the pipeline crash-proof.** Running without an API key, or with an exhausted budget, still produces structurally valid pages. The CI step doesn't fail; it just produces lower-quality content. I can re-run the pipeline later with a working key and the cache miss detection handles content upgrades automatically. **pair_slug as the dedup key is clean.** No separate "already processed" log file, no state outside the database. If the row exists, skip. If not, generate. This is the same pattern I use for the HuggingFace model fetch ETL — the database table is the state. **Pipeline-tag grouping produces coherent pairs.** The comparisons are topically relevant by construction. A visitor on an `image-to-image` model page sees comparisons to other `image-to-image` models, not cross-category noise. ## What I'd do differently **Pure download-count ranking picks the boring pairs.** The top-downloaded models in each pipeline tag are often the most generic: `bert-base-uncased`, `gpt2`, tutorial-tier checkpoints. The interesting comparisons for a developer with a real task are often between models ranked 5th and 12th in a category — specific enough to have real architectural differences, popular enough to have community benchmarks. I'd add a recency weight (something like `recent_30d_downloads / total_downloads`) to surface models that are gaining traction rather than coasting on historical popularity. **Sequential generation is slow.** Right now pairs run one at a time. With 50 pairs per run and ~1-2 seconds per Haiku call, that's a 1-2 minute sequential run. Async batching at 5-10 concurrent calls with a rate limiter would cut that to 10-20 seconds. I haven't prioritized this because the run happens in a CI job that's off the critical path, but it would make iteration faster. **The prompt fields are too open-ended.** Asking Claude to list "differences" with no axis constraints produces inconsistent comparisons — one pair gets architecture differences, another gets license differences, another gets vague use-case differences. The output isn't comparable across pairs. I'd specify the axes explicitly: "List differences specifically in: model architecture, license, parameter count, supported languages, inference cost." **Comparison pages aren't linked from model detail pages yet.** A visitor on the Llama 3.2 page should see "Compare Llama 3.2 with:" and a list of its cached comparisons. I have the data to build this; the UI just isn't wired. I'm adding it next. ## FAQ **How many comparison pages are there now?** I don't have an exact count to publish yet — I'll run a `SELECT COUNT(*) FROM model_compare` and post an update in 30 days. The theoretical cap from the pairing logic is around 180 active pairs. **Does Claude ever refuse to compare two models?** Not in practice. The prompt is framed as "produce a developer-focused comparison JSON," which is a concrete technical task. The closest failure mode is Haiku producing generic differences when both models are obscure fine-tunes with minimal distinguishing metadata — technically valid output, not particularly useful. **What happens when a model gets deleted from HuggingFace?** The comparison row stays in the database. The compare page stays live with stale content. I don't have a cleanup pipeline for deleted models yet. The public HuggingFace API doesn't expose a "deleted models" feed, so detection requires checking 404 responses during the nightly model refresh — something I'll add once I have a reason to care about page count accuracy. **Why not use embedding similarity to pick the most relevant models to compare?** Budget and complexity. Embedding 1,500 model summaries and running similarity queries needs either a vector extension (Turso has pgvector) or an external store. For a site with effectively zero traffic today, that infrastructure overhead isn't justified. The pipeline_tag grouping achieves "plausibly relevant pairs" with zero extra infrastructure cost, and I can upgrade the pairing logic later without changing the generation or caching layers. ## Related articles * Three JSON extraction patterns I use with Claude Haiku in production directory ETLs * How I fixed the slug collision that silenced aiappdex.com for 36 hours _Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted._

A SQL database in Rust: SQLite-compatible, now also speaking Postgres (experimental). The LLVM of databases. - tursodatabase/turso

Two compounding bugs — slugify() normalizing distinct HF model IDs to the same string, and @libsql/client changing error shape between versions — silenced the directory. Diagnosis and code.

Use Turso MCP with Claude or Cursor to manage your edge databases, provision libSQL instances, and rotate security tokens via your AI agent.

Automatic admin interface for Drizzle Schema using Nuxt - awecode/autoadmin

Control Turso databases and edge locations conversationally. Provision, secure, and audit your serverless SQLite infrastructure with the Turso MCP.

Control Turso databases and edge locations conversationally. Provision, secure, and audit your serverless SQLite infrastructure with the Turso MCP.