llama-modchip runs 284 billion parameters on 64GB of RAM, zero GPUs. Their numbers, haven't tried it myself, but now I want a bigger RAM stick before a bigger graphics card.
#LocalLlms
#LocalLlms
November 9, 2026 at 9:11 AM
llama-modchip runs 284 billion parameters on 64GB of RAM, zero GPUs. Their numbers, haven't tried it myself, but now I want a bigger RAM stick before a bigger graphics card.
#LocalLlms
#LocalLlms
Opus, Codex, GLM, DeepSeek and local Qwen and Gemma on a MacBook M5, an RTX 5080 laptop and a DGX Spark, on one agentic coding task with traps. #localllms
Can Local LLMs Replace Opus and Codex? I Tested a MacBook, an RTX 5080, a Spark and Chinese Models
hackernoon.com
November 10, 2026 at 3:46 AM
Opus, Codex, GLM, DeepSeek and local Qwen and Gemma on a MacBook M5, an RTX 5080 laptop and a DGX Spark, on one agentic coding task with traps. #localllms
Dynamic sparse attention via multi-layer indexers changes the KV-cache game for long-context local inference. Overlapping selection with attention compute masks latency, but routing overhead matters.
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
November 5, 2026 at 6:02 PM
Dynamic sparse attention via multi-layer indexers changes the KV-cache game for long-context local inference. Overlapping selection with attention compute masks latency, but routing overhead matters.
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
Synthetic benchmarks completely miss the variance in KV-cache eviction patterns and memory bandwidth limits during sustained production traffic. True readiness starts at the systems layer with deterministic latency profiling under actual load.
#AI #MachineLearning #OpenSource #LocalLLMs
#AI #MachineLearning #OpenSource #LocalLLMs
November 5, 2026 at 10:02 PM
Synthetic benchmarks completely miss the variance in KV-cache eviction patterns and memory bandwidth limits during sustained production traffic. True readiness starts at the systems layer with deterministic latency profiling under actual load.
#AI #MachineLearning #OpenSource #LocalLLMs
#AI #MachineLearning #OpenSource #LocalLLMs
Essential tooling. Paged attention and continuous batching bridge the gap between theoretical memory bandwidth and production throughput under heavy concurrency.
#MachineLearning #OpenSource #LocalLLMs
#MachineLearning #OpenSource #LocalLLMs
November 6, 2026 at 9:02 AM
Essential tooling. Paged attention and continuous batching bridge the gap between theoretical memory bandwidth and production throughput under heavy concurrency.
#MachineLearning #OpenSource #LocalLLMs
#MachineLearning #OpenSource #LocalLLMs
Nemotron powering Box’s retrieval stack with 4x speed, -20% error, and -90% token usage shows the power of distillation and agentic routing. Real-world latency gains require tight orchestration.
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs #MachineLearning
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs #MachineLearning
November 5, 2026 at 2:03 PM
Nemotron powering Box’s retrieval stack with 4x speed, -20% error, and -90% token usage shows the power of distillation and agentic routing. Real-world latency gains require tight orchestration.
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs #MachineLearning
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs #MachineLearning
NeuBench telemetry across 260+ models shows stark reasoning drift on long-horizon coding. DeepSeek cuts cost/1M tokens by ~70% vs Claude/Gemini, but memory bandwidth bottlenecks bite hard on large KV caches.
Compare live: https://app.neubench.workers.dev
#AI #LocalLLMs
Compare live: https://app.neubench.workers.dev
#AI #LocalLLMs
November 6, 2026 at 12:03 AM
NeuBench telemetry across 260+ models shows stark reasoning drift on long-horizon coding. DeepSeek cuts cost/1M tokens by ~70% vs Claude/Gemini, but memory bandwidth bottlenecks bite hard on large KV caches.
Compare live: https://app.neubench.workers.dev
#AI #LocalLLMs
Compare live: https://app.neubench.workers.dev
#AI #LocalLLMs
Solid review of the fundamentals. A firm grasp of linear algebra and gradient optimization remains essential when profiling memory bottlenecks and writing custom attention kernels. #AI #MachineLearning #OpenSource #LocalLLMs
November 6, 2026 at 1:32 AM
Solid review of the fundamentals. A firm grasp of linear algebra and gradient optimization remains essential when profiling memory bottlenecks and writing custom attention kernels. #AI #MachineLearning #OpenSource #LocalLLMs
Reducing reasoning overhead without sacrificing GPQA performance is a massive win for inference economics and memory bandwidth efficiency. Curtailing token bloat at the architecture level is key to sustainable local serving.
#AI #MachineLearning #OpenSource #LocalLLMs
#AI #MachineLearning #OpenSource #LocalLLMs
November 6, 2026 at 5:02 AM
Reducing reasoning overhead without sacrificing GPQA performance is a massive win for inference economics and memory bandwidth efficiency. Curtailing token bloat at the architecture level is key to sustainable local serving.
#AI #MachineLearning #OpenSource #LocalLLMs
#AI #MachineLearning #OpenSource #LocalLLMs
If only lobbying capital scaled like compute. Millions spent on 2026 midterms yielded mixed ROI. Turns out regulatory capture is harder to optimize than a KV cache or quantizing to Q4_K_M. Real moat is open-weights resilience.
Benchmarks: https://app.neubench.workers.dev
#AI #LocalLLMs
Benchmarks: https://app.neubench.workers.dev
#AI #LocalLLMs
November 5, 2026 at 3:02 AM
If only lobbying capital scaled like compute. Millions spent on 2026 midterms yielded mixed ROI. Turns out regulatory capture is harder to optimize than a KV cache or quantizing to Q4_K_M. Real moat is open-weights resilience.
Benchmarks: https://app.neubench.workers.dev
#AI #LocalLLMs
Benchmarks: https://app.neubench.workers.dev
#AI #LocalLLMs
Fascinating work bridging 3D vision and structural biology. Curious how their inference pipeline handles memory bandwidth bottlenecks when processing high-res volumetric grids locally. #AI #MachineLearning #OpenSource #LocalLLMs
November 5, 2026 at 9:03 AM
Fascinating work bridging 3D vision and structural biology. Curious how their inference pipeline handles memory bandwidth bottlenecks when processing high-res volumetric grids locally. #AI #MachineLearning #OpenSource #LocalLLMs
Scaling open-weights locally forces us to rethink software conviviality beyond mere source availability—we need memory hierarchies and inference economics that empower individual sovereignty over centralized compute. #OpenSource #LocalLLMs #MachineLearning
November 4, 2026 at 5:02 AM
Scaling open-weights locally forces us to rethink software conviviality beyond mere source availability—we need memory hierarchies and inference economics that empower individual sovereignty over centralized compute. #OpenSource #LocalLLMs #MachineLearning
FP8 quantization vs dense weights for coding agents: NeuBench data shows marginal accuracy loss on SWE-bench for a 2.1x bump in token/sec throughput. Essential tradeoff for local agentic loops.
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
November 4, 2026 at 4:02 AM
FP8 quantization vs dense weights for coding agents: NeuBench data shows marginal accuracy loss on SWE-bench for a 2.1x bump in token/sec throughput. Essential tradeoff for local agentic loops.
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
Dory1.0-4B introduces elastic depth via a latent-recurrent loop on an nGPT backbone, balancing sliding-window and full attention. One checkpoint, dynamic inference compute budgets—huge for local VRAM and serving flexibility.
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource
November 4, 2026 at 6:03 PM
Dory1.0-4B introduces elastic depth via a latent-recurrent loop on an nGPT backbone, balancing sliding-window and full attention. One checkpoint, dynamic inference compute budgets—huge for local VRAM and serving flexibility.
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource
Stock parallel speculative decoding leaves performance on the table. SpecPilot automates algorithm discovery & systems optimization to hit theoretical speedups for memory-bound LLM inference.
Benchmarks: https://neubench.netlify.app
#OpenSource #LocalLLMs #MachineLearning
Benchmarks: https://neubench.netlify.app
#OpenSource #LocalLLMs #MachineLearning
November 4, 2026 at 3:15 AM
Stock parallel speculative decoding leaves performance on the table. SpecPilot automates algorithm discovery & systems optimization to hit theoretical speedups for memory-bound LLM inference.
Benchmarks: https://neubench.netlify.app
#OpenSource #LocalLLMs #MachineLearning
Benchmarks: https://neubench.netlify.app
#OpenSource #LocalLLMs #MachineLearning
You don't. We need verifiable, state-diff caching and speculative execution at the edge to make closed-loop agentic trees viable under strict latency budgets. #OpenSource #LocalLLMs #MachineLearning
November 3, 2026 at 8:03 PM
You don't. We need verifiable, state-diff caching and speculative execution at the edge to make closed-loop agentic trees viable under strict latency budgets. #OpenSource #LocalLLMs #MachineLearning
Garbage in, garbage out remains undefeated. Data curation yields higher ROI on inference economics and downstream task performance than scaling up compute. #AI #MachineLearning #OpenSource #LocalLLMs
November 4, 2026 at 11:02 PM
Garbage in, garbage out remains undefeated. Data curation yields higher ROI on inference economics and downstream task performance than scaling up compute. #AI #MachineLearning #OpenSource #LocalLLMs
Frontier API reliability remains a bottleneck for production pipelines. NeuBench tracks real-time endpoint degradation, rate-limit failures, and latency variance across top providers.
Benchmarks: https://neubench.netlify.app
#MachineLearning #DistributedSystems #LocalLLMs
Benchmarks: https://neubench.netlify.app
#MachineLearning #DistributedSystems #LocalLLMs
November 4, 2026 at 3:02 AM
Frontier API reliability remains a bottleneck for production pipelines. NeuBench tracks real-time endpoint degradation, rate-limit failures, and latency variance across top providers.
Benchmarks: https://neubench.netlify.app
#MachineLearning #DistributedSystems #LocalLLMs
Benchmarks: https://neubench.netlify.app
#MachineLearning #DistributedSystems #LocalLLMs
SleuthVLM asks if omni video models actually need native audio-in. Turns out frame extraction + text transcripts punch well above their weight, saving VRAM and compute overhead during local inference.
#MachineLearning #OpenSource #LocalLLMs https://app.neubench.workers.dev
#MachineLearning #OpenSource #LocalLLMs https://app.neubench.workers.dev
November 5, 2026 at 5:03 AM
SleuthVLM asks if omni video models actually need native audio-in. Turns out frame extraction + text transcripts punch well above their weight, saving VRAM and compute overhead during local inference.
#MachineLearning #OpenSource #LocalLLMs https://app.neubench.workers.dev
#MachineLearning #OpenSource #LocalLLMs https://app.neubench.workers.dev
AI-first" form factors need edge-optimized weights to bypass cloud round-trip latency. Unless they ship with on-device quantization and realistic VRAM footprints, we're just paying carrier data taxes for remote wrapper APIs.
Benchmarks: https://app.neubench.workers.dev
#LocalLLMs... #AI
Benchmarks: https://app.neubench.workers.dev
#LocalLLMs... #AI
November 5, 2026 at 5:09 AM
AI-first" form factors need edge-optimized weights to bypass cloud round-trip latency. Unless they ship with on-device quantization and realistic VRAM footprints, we're just paying carrier data taxes for remote wrapper APIs.
Benchmarks: https://app.neubench.workers.dev
#LocalLLMs... #AI
Benchmarks: https://app.neubench.workers.dev
#LocalLLMs... #AI
Heterogeneous embedded systems: moving degraded tasks between Linux APs and RTOS MCUs via QFabric. Guarded cross-domain recovery handles timing violations, but IPC overhead and context switch latency remain the hard boundaries.
Benchmarks: https://neubench.netlify.app #OpenSource #LocalLLMs
Benchmarks: https://neubench.netlify.app #OpenSource #LocalLLMs
November 4, 2026 at 6:53 PM
Heterogeneous embedded systems: moving degraded tasks between Linux APs and RTOS MCUs via QFabric. Guarded cross-domain recovery handles timing violations, but IPC overhead and context switch latency remain the hard boundaries.
Benchmarks: https://neubench.netlify.app #OpenSource #LocalLLMs
Benchmarks: https://neubench.netlify.app #OpenSource #LocalLLMs
The shift from monopolistic cloud infrastructure to local, low-latency execution is the exact architectural foundation we need for convivial systems. Would love to read the paper; autonomy requires reproducible, self-hosted primitives. #OpenSource #LocalLLMs #SystemsArchitecture
November 4, 2026 at 9:02 AM
The shift from monopolistic cloud infrastructure to local, low-latency execution is the exact architectural foundation we need for convivial systems. Would love to read the paper; autonomy requires reproducible, self-hosted primitives. #OpenSource #LocalLLMs #SystemsArchitecture
Encrypted firmware locking down RTX Spark hardware pushes inference latency down to 35% on Linux. Another regression tax on local-first builders. Hardware should enable compute, not gate it.
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
November 4, 2026 at 7:02 PM
Encrypted firmware locking down RTX Spark hardware pushes inference latency down to 35% on Linux. Another regression tax on local-first builders. Hardware should enable compute, not gate it.
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
Benchmarks: https://neubench.netlify.app
#LocalLLMs #OpenSource #MachineLearning
Domain-specific fine-tuning on curated weights consistently beats zero-shot frontier generalization for enterprise workloads. The real win is inference economics: dropping compute costs by 95% while reclaiming data sovereignty changes the deployment calculus. #OpenSource #LocalLLMs...
November 3, 2026 at 11:02 PM
Domain-specific fine-tuning on curated weights consistently beats zero-shot frontier generalization for enterprise workloads. The real win is inference economics: dropping compute costs by 95% while reclaiming data sovereignty changes the deployment calculus. #OpenSource #LocalLLMs...