Standardization can often feel opaque from the outside, so we are organizing this informal, community-driven gathering right after the official GPU for the Web Working Group event.

El CSP de Manifest V3 bloquea el loader WASM de ONNX por defecto. Así lo resolvió AWFlow para correr LLMs locales sin API key de OpenAI.

Verified 2026 guide to running local LLMs offline on Android and iPhone — best apps, built-in models (Gemini Nano, Apple Foundation Models), WebLLM in the browser, which small models fit phone RAM, re...

High-performance In-browser LLM Inference Engine . Contribute to mlc-ai/web-llm development by creating an account on GitHub.

Posted by stephenlblum - 1 vote and 0 comments

In the era of cloud-hosted AI, we’ve become comfortable sending our most sensitive data to remote servers. But when it comes to medical queries—like checking for **drug-to-drug interactions** —privacy isn't just a feature; it's a human right. 🛡️ With the recent explosion of **WebGPU AI** and the maturation of **local LLMs** , we can finally move the "brain" of our applications directly into the user's browser. In this tutorial, we are building a high-performance, **browser-based AI** tool that uses **WebLLM** and **WebGPU** to perform millisecond-level drug compatibility checks. No data ever leaves the device, ensuring 100% data residency and lightning-fast **edge computing** performance. ## The Architecture: Why WebGPU? Traditionally, running a Large Language Model (LLM) required a massive Python backend with expensive GPUs. **WebGPU** changes the game by providing low-level access to the local graphics card directly from the browser. **WebLLM** leverages this to run models like Llama-3 or Mistral in the browser sandbox. ### System Data Flow graph TD UserInput[User Inputs Medications] -->|React State| Engine[WebLLM Engine Instance] Engine -->|Compute Shaders| WebGPU[WebGPU API] WebGPU -->|Parallel Processing| LocalGPU[Device VRAM/GPU] LocalGPU -->|Token Generation| Engine Engine -->|Streamed Response| UI[React Frontend Display] subgraph Browser_Sandbox Engine WebGPU UI end subgraph Privacy_Boundary Browser_Sandbox end ExternalServer((Cloud / Internet)) -.->|Data Never Sent| Privacy_Boundary ## Prerequisites 🛠️ To follow this advanced guide, you'll need: * **Tech Stack** : React (v18+), TypeScript, Vite. * **Library** : `@mlc-ai/web-llm`. * **Hardware** : A GPU supporting WebGPU (Latest Chrome/Edge/Arc). ## Step 1: Initializing the WebLLM Engine First, we need to create a singleton or a hook to manage our AI engine. Since loading a model (~2GB-5GB) takes time, we need to handle the progress state effectively. // useWebLLM.ts import { useState, useEffect } from "react"; import * as webllm from "@mlc-ai/web-llm"; export function useWebLLM(modelId: string) { const [engine, setEngine] = useState<webllm.MLCEngine | null>(null); const [loadingProgress, setLoadingProgress] = useState(0); const initEngine = async () => { const engineInstance = new webllm.MLCEngine(); // Callback to track model download/loading progress engineInstance.setInitProgressCallback((report) => { setLoadingProgress(Math.round(report.progress * 100)); }); await engineInstance.reload(modelId); setEngine(engineInstance); }; return { engine, loadingProgress, initEngine }; } ## Step 2: Crafting the Interaction Logic Drug interaction retrieval requires a high degree of accuracy. We will use a structured system prompt to ensure the LLM acts as a clinical pharmacist. const SYSTEM_PROMPT = ` You are a clinical pharmacy expert. Your task is to analyze two or more medications and identify potential drug-to-drug interactions. Provide the output in the following format: 1. Interaction Level (Mild, Moderate, Severe) 2. Mechanism of Action 3. Recommendation Be concise and stick to clinical facts. If no interaction is found, state so. `; const checkInteractions = async (engine: webllm.MLCEngine, drugs: string[]) => { const userPrompt = `Check interactions for: ${drugs.join(", ")}`; const messages: webllm.ChatCompletionMessageParam[] = [ { role: "system", content: SYSTEM_PROMPT }, { role: "user", content: userPrompt } ]; const chunks = await engine.chat.completions.create({ messages, stream: true, // We want that sweet typewriter effect! }); return chunks; }; ## Step 3: The React UI Layer We want a clean, professional interface that handles the streaming response and gives users confidence. import React, { useState } from 'react'; import { useWebLLM } from './hooks/useWebLLM'; const DrugChecker = () => { const { engine, loadingProgress, initEngine } = useWebLLM("Llama-3-8B-Instruct-v0.1-q4f16_1-MLC"); const [drugs, setDrugs] = useState(""); const [result, setResult] = useState(""); const handleConsult = async () => { if (!engine) return; setResult(""); // Clear previous const stream = await checkInteractions(engine, drugs.split(",")); for await (const chunk of stream) { const content = chunk.choices[0]?.delta?.content || ""; setResult((prev) => prev + content); } }; if (!engine) { return ( <div className="p-10 text-center"> <button onClick={initEngine} className="bg-blue-600 text-white px-6 py-2 rounded"> Initialize Secure Local AI </button> <p className="mt-4">Loading Model: {loadingProgress}%</p> </div> ); } return ( <div className="max-w-2xl mx-auto p-6"> <h2 className="text-2xl font-bold mb-4">🛡️ Local Drug Interaction Checker</h2> <textarea className="w-full border p-3 rounded mb-4" placeholder="Enter medications (e.g., Warfarin, Aspirin)..." onChange={(e) => setDrugs(e.target.value)} /> <button onClick={handleConsult} className="bg-green-600 text-white px-8 py-3 rounded-lg font-semibold" > Analyze Privately </button> <div className="mt-8 p-4 bg-gray-50 rounded border whitespace-pre-wrap"> {result || "Analysis will appear here..."} </div> </div> ); }; ## The "Official" Way 🥑 Building a proof-of-concept is easy, but making **Edge AI** production-ready involves handling model caching, VRAM memory management, and specialized RAG (Retrieval-Augmented Generation) architectures to ensure medical data is up to date. For more advanced patterns on optimizing WebGPU shaders, managing local vector databases, and high-performance AI deployment strategies, I highly recommend checking out the **WellAlly Technology Blog**. It's the source of inspiration for this build and contains deeper dives into "Local-First" software engineering. ## Performance & Privacy Trade-offs 📊 * **Latency** : Once the model is loaded into the browser cache (IndexedDB), the "Time to First Token" is often faster than a round-trip to OpenAI's servers. * **Cost** : Your server costs drop to zero. The user provides the compute. 💸 * **Security** : Since there is no backend API for the drug query, there is no log to breach. This is the ultimate **HIPAA-compliant** architecture by design. ## Conclusion The browser is no longer just a document viewer; it's a powerful AI execution environment. By combining **WebGPU** and **WebLLM** , we can build tools that were impossible a year ago. **Next Steps** : 1. Try adding a local vector store (like Voy) to allow the AI to search through updated FDA documentation locally. 2. Implement model-switching logic to use smaller models for simpler queries. What are you planning to build with WebGPU? Let me know in the comments below! 👇

An open-source project runs full LLMs entirely inside your browser tab, no server or API call needed.

High-performance In-browser LLM Inference Engine . Contribute to mlc-ai/web-llm development by creating an account on GitHub.

Local Execution Like Never Before! The Mind-Blowing Speed of "WebLLM" Powered by WebGPU

High-performance In-browser LLM Inference Engine . Contribute to mlc-ai/web-llm development by creating an account on GitHub.

High-performance In-browser LLM Inference Engine . Contribute to mlc-ai/web-llm development by creating an account on GitHub.

High-performance In-browser LLM Inference Engine . Contribute to mlc-ai/web-llm development by creating an account on GitHub.