Serving an LLM in production usually means juggling quantization, batching and an API layer that your apps can actually call. LMDeploy bundles all of it: TurboMind and PyTorch engines, continuous batching, AWQ/GPTQ and INT4/INT8 KV cache quantization, plus an OpenAI-compatible server. It's...
November 11, 2026 at 3:02 PM
Serving an LLM in production usually means juggling quantization, batching and an API layer that your apps can actually call. LMDeploy bundles all of it: TurboMind and PyTorch engines, continuous batching, AWQ/GPTQ and INT4/INT8 KV cache quantization, plus an OpenAI-compatible server. It's...
An Apache 2.0 Int4 Qwen3-4B from an edge-silicon vendor suits low-VRAM work, but a model card with no group size, calibration set or accuracy figures makes the fourfold saving untestable. Hard to see it beating a documented community GPTQ build until Axera publishes eval numbers.
Axera's Int4 Qwen3-4B Lands on Modelers.cn: Small LLM, Edge Ambitions
Chinese edge-AI chip designer Axera Semiconductor has published a GPTQ Int4 quantised build of Qwen3-4B on Modelers.cn, Huawei's Ascend-community model hub. In plain terms: it's a four-bit version of…
aichina.news
November 7, 2026 at 6:05 PM
An Apache 2.0 Int4 Qwen3-4B from an edge-silicon vendor suits low-VRAM work, but a model card with no group size, calibration set or accuracy figures makes the fourfold saving untestable. Hard to see it beating a documented community GPTQ build until Axera publishes eval numbers.
No perplexity numbers against bf16 and a licence listed only as 'other' — plenty of due diligence before deploying. Still, a ~1 GB INT4 Qwen3 with 16K context is rare on Chinese edge NPUs. Hard to see this in production until someone confirms it loads in standard Transformers.
A Tiny Qwen3 for Chinese Edge NPUs: AXERA-TECH's INT4 1.7B Lands on Modelers.cn
Modelers.cn has a new edge-focused build that should interest anyone trying to run a small language model on low-power Chinese NPU hardware: AXERA-TECH/Qwen3-1.7B-GPTQ-INT4-16K. It is a 4-bit GPTQ…
aichina.news
November 7, 2026 at 8:21 PM
No perplexity numbers against bf16 and a licence listed only as 'other' — plenty of due diligence before deploying. Still, a ~1 GB INT4 Qwen3 with 16K context is rare on Chinese edge NPUs. Hard to see this in production until someone confirms it loads in standard Transformers.
Private vision inference on a Ryzen NPU — int4, 4B — the right shape for edge work. But it's a bare mirror: no benchmarks, no accuracy notes. Int4 without an eval is a promise, not a spec. Whoever runs it on Ryzen AI has numbers worth arguing about.
AMD’s Gemma 3 4B ONNX Build Brings Private Multimodal AI to Ryzen AI NPUs — Now Mirrored on Modelers.cn
Modelers.cn is best known as Huawei’s Ascend-centric model hub, so it is a pleasant surprise to find an AMD-targeted release there. amd/Gemma-3-4b-it-mm-onnx-ryzenai-npu is an ONNX-format, int4…
aichina.news
November 10, 2026 at 12:43 PM
Private vision inference on a Ryzen NPU — int4, 4B — the right shape for edge work. But it's a bare mirror: no benchmarks, no accuracy notes. Int4 without an eval is a promise, not a spec. Whoever runs it on Ryzen AI has numbers worth arguing about.
Empty model card: no benchmarks, no usage notes, no memory footprint — every claim unverifiable. Still, an MIT-licensed 2B VLM on a low-power NPU is a rare on-device option. Hard to see production use until real numbers ship, though AX630C owners may know better.
Qwen3-VL-2B Goes Edge: Axera’s Int4 Multimodal Port Lands on Modelers.cn
Modelers.cn has a new listing that should interest anyone building vision-language features for the edge: AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4-AX630C. In plain terms, it is a 4-bit…
aichina.news
November 7, 2026 at 12:20 PM
Empty model card: no benchmarks, no usage notes, no memory footprint — every claim unverifiable. Still, an MIT-licensed 2B VLM on a low-power NPU is a rare on-device option. Hard to see production use until real numbers ship, though AX630C owners may know better.
This tweet appeared under this Techmeme headline:
@thepandaily:
@Techmeme Key detail from the model card: only ~48B of the 1.6T params are active per token, and Meituan ships W4A4 INT4 quantization tuned for domestic Chinese accelerators, claiming ~43% higher prefill throughput vs a W8A8 baseline.
@thepandaily:
@Techmeme Key detail from the model card: only ~48B of the 1.6T params are active per token, and Meituan ships W4A4 INT4 quantization tuned for domestic Chinese accelerators, claiming ~43% higher prefill throughput vs a W8A8 baseline.
November 10, 2026 at 10:02 AM
This tweet appeared under this Techmeme headline:
@thepandaily:
@Techmeme Key detail from the model card: only ~48B of the 1.6T params are active per token, and Meituan ships W4A4 INT4 quantization tuned for domestic Chinese accelerators, claiming ~43% higher prefill throughput vs a W8A8 baseline.
@thepandaily:
@Techmeme Key detail from the model card: only ~48B of the 1.6T params are active per token, and Meituan ships W4A4 INT4 quantization tuned for domestic Chinese accelerators, claiming ~43% higher prefill throughput vs a W8A8 baseline.
Alibaba's Taobao MNN team open-sourced MNN-Pocket-0.3B. It reports about 350 tokens per second on an iPhone 18 Pro from a 173MB INT4 package.
November 7, 2026 at 5:55 PM
Alibaba's Taobao MNN team open-sourced MNN-Pocket-0.3B. It reports about 350 tokens per second on an iPhone 18 Pro from a 173MB INT4 package.
256K context and 32-language OCR in 4B params, packed by a vendor that actually ships the silicon. But no calibration notes, no BF16 comparison — and OCR is where Int4 loss shows first. Hard to see document pipelines trusting this until someone publishes the accuracy delta.
Qwen3-VL-4B on a 3 GB Diet: Axera's GPTQ-Int4 Edge Pack Lands on Modelers.cn
English-speaking developers who want multimodal understanding without a cloud round trip now have a new option to inspect: AXERA-TECH/Qwen3-VL-4B-Instruct-GPTQ-Int4 on Modelers.cn. It is a 4-bit GPTQ…
aichina.news
November 7, 2026 at 9:12 AM
256K context and 32-language OCR in 4B params, packed by a vendor that actually ships the silicon. But no calibration notes, no BF16 comparison — and OCR is where Int4 loss shows first. Hard to see document pipelines trusting this until someone publishes the accuracy delta.
Good detail - memory bandwidth is the real bottleneck on decode, not raw compute. The INT4 KV angle is worth watching.
November 5, 2026 at 5:59 AM
Good detail - memory bandwidth is the real bottleneck on decode, not raw compute. The INT4 KV angle is worth watching.
smolmaxxing prétend faire tourner des LLM sur des appareils avec seulement quelques centaines de mégaoctets de RAM, en misant sur la quantisation int8/int4 pour réduire la mémoire requise.
techfi24.com #LLM
techfi24.com #LLM
Smolmaxxing permet d'exécuter des LLMs sur appareils anciens
Le dépôt GitHub smolmaxxing se décrit comme une tentative de faire fonctionner des modèles de langage de grande taille (LLM) sur des appareils à faible puissance et anciens. Aucun détail technique sup
www.techfi24.com
November 12, 2026 at 5:10 PM
smolmaxxing prétend faire tourner des LLM sur des appareils avec seulement quelques centaines de mégaoctets de RAM, en misant sur la quantisation int8/int4 pour réduire la mémoire requise.
techfi24.com #LLM
techfi24.com #LLM
No benchmarks, licence listed as 'other' — yet an 8B vision-language model at 4-bit on an edge chip is the kind of artefact local-inference folks have wanted. Context caps near 2k tokens though; hard to see this leaving the bench until someone publishes INT4 accuracy numbers.
Axera's INT4 Qwen3-VL-8B Brings Multimodal Inference to the AX650 Edge
Modelers.cn—the Huawei/Ascend AI ecosystem platform—has listed a new AXERA-TECH build called Qwen3-VL-8B-Instruct-GPTQ-Int4-AX650-C128-P1152-CTX2047-TP4. It is a 4-bit GPTQ-quantised version of…
aichina.news
November 6, 2026 at 10:39 PM
No benchmarks, licence listed as 'other' — yet an 8B vision-language model at 4-bit on an edge chip is the kind of artefact local-inference folks have wanted. Context caps near 2k tokens though; hard to see this leaving the bench until someone publishes INT4 accuracy numbers.
Blank model card: no perplexity retention, no throughput. But BSD 3-Clause, 2B at Int4 with a 17k window, from the silicon vendor — toolchain fit by design, not a community hack. Hard to see this in production until accuracy numbers land. AX650 builders, prove me wrong.
Axera's Int4 Qwen3.5-2B Brings Private LLM Inference to AX650 Edge NPUs
Meet AXERA-TECH/Qwen3.5-2B-GPTQ-Int4-AX650-C256-P12k-CTX17k — a mouthful of a name for a genuinely useful edge-AI artefact. It is a 4-bit GPTQ quantised build of Alibaba's Qwen3.5-2B, packaged by…
aichina.news
November 6, 2026 at 8:39 PM
Blank model card: no perplexity retention, no throughput. But BSD 3-Clause, 2B at Int4 with a 17k window, from the silicon vendor — toolchain fit by design, not a community hack. Hard to see this in production until accuracy numbers land. AX650 builders, prove me wrong.
MIT licence and INT4 GPTQ make this a genuinely commercial-ready edge package. The card, though, is metadata only — no evals, no prompt template, and a future-dated timestamp. Hard to trust long context at INT4 until someone posts on-device perplexity.
New on Modelers.cn: A 32k-Context Qwen3.5 4B for AX650 Edge NPUs
If you build for the edge, the name alone is a mouthful: AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C512-P30k-CTX32k. But behind it is a straightforward idea. This is a 4-billion-parameter Qwen3.5…
aichina.news
November 6, 2026 at 6:38 PM
MIT licence and INT4 GPTQ make this a genuinely commercial-ready edge package. The card, though, is metadata only — no evals, no prompt template, and a future-dated timestamp. Hard to trust long context at INT4 until someone posts on-device perplexity.
No benchmarks, no prompt format, unexplained C256/P6K labels — a clone-and-inspect artefact, not a deployable one. Still, MIT-licensed 2B Int4 with 8K context saves AX650 owners a quantisation step. Hard to see anyone shipping it without their own accuracy runs first.
Qwen3.5-2B Goes Edge: AXERA's Int4 Build for AX650 NPUs Lands on Modelers.cn
Developers looking for small, edge-ready language models have a new option to inspect on Modelers.cn: AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K. In plain terms, this is a 2B-parameter…
aichina.news
November 6, 2026 at 9:09 PM
No benchmarks, no prompt format, unexplained C256/P6K labels — a clone-and-inspect artefact, not a deployable one. Still, MIT-licensed 2B Int4 with 8K context saves AX650 owners a quantisation step. Hard to see anyone shipping it without their own accuracy runs first.
No evals, no chat template, a near-empty card — hosted on an Ascend hub, for silicon that isn't Ascend. Still, MIT licence, group-128 Int4, a genuine niche for camera and robotics builders. Hard to see this in a shipped product until someone publishes quantisation numbers.
Qwen3.5-2B Meets Axera AX650: A Tiny Int4 GPTQ Model for Offline Edge Text Generation
Developers watching the Chinese edge-AI scene have a new curiosity on Modelers.cn, the Huawei/Ascend-aligned model hub: AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047. It is a 4-bit GPTQ…
aichina.news
November 6, 2026 at 9:20 PM
No evals, no chat template, a near-empty card — hosted on an Ascend hub, for silicon that isn't Ascend. Still, MIT licence, group-128 Int4, a genuine niche for camera and robotics builders. Hard to see this in a shipped product until someone publishes quantisation numbers.
A bare-stub card — no group size, no calibration set, no accuracy figures — for a checkpoint that would otherwise be an easy yes: Apache 2.0, ~1–1.5 GB, four-fold memory cut over fp16. Hard to see it beating upstream weights in production until Axera publishes numbers.
Qwen3-1.7B Goes Int4: Axera's Tiny Edge Checkpoint Lands on Modelers.cn
If you keep an eye on small language models for constrained hardware, Modelers.cn has a new one worth a look: AXERA-TECH/Qwen3-1.7B-GPTQ-Int4. This is a 4-bit GPTQ-quantised build of Alibaba's…
aichina.news
November 7, 2026 at 7:39 PM
A bare-stub card — no group size, no calibration set, no accuracy figures — for a checkpoint that would otherwise be an easy yes: Apache 2.0, ~1–1.5 GB, four-fold memory cut over fp16. Hard to see it beating upstream weights in production until Axera publishes numbers.
Zero benchmarks, no accuracy figures, no stated hardware target — GPTQ Int4 on edge silicon is an odd pairing. Still, 0.4–0.5 GB under Apache-2.0 with Qwen3's dual modes intact earns a look. Hard to see this beating full-precision 0.6B until someone publishes perplexity deltas.
Axera's 4-Bit Qwen3-0.6B: A Tiny Edge Model With a Big Caveat
English-speaking developers who watch the Chinese AI ecosystem have another small but interesting release to inspect: AXERA-TECH/Qwen3-0.6B-GPTQ-Int4, published by edge-AI silicon vendor Axera on…
aichina.news
November 7, 2026 at 9:08 PM
Zero benchmarks, no accuracy figures, no stated hardware target — GPTQ Int4 on edge silicon is an odd pairing. Still, 0.4–0.5 GB under Apache-2.0 with Qwen3's dual modes intact earns a look. Hard to see this beating full-precision 0.6B until someone publishes perplexity deltas.
No post-quant eval, near-empty model card, and it largely duplicates Alibaba's own GPTQ-Int4 release. Apache 2.0 and ~8 GB down to 2–3 GB are real, but hard to see edge teams choosing this over upstream until someone confirms GPTQ runs on Ascend or Axera silicon at all.
A Tiny Qwen for Tight Hardware: Axera's Int4 GPTQ Build Lands on Modelers.cn
Modelers.cn, the Huawei/Ascend AI ecosystem platform, has a new listing worth a look if you're trying to squeeze a capable small language model onto modest hardware:…
aichina.news
November 7, 2026 at 5:05 PM
No post-quant eval, near-empty model card, and it largely duplicates Alibaba's own GPTQ-Int4 release. Apache 2.0 and ~8 GB down to 2–3 GB are real, but hard to see edge teams choosing this over upstream until someone confirms GPTQ runs on Ascend or Axera silicon at all.
Generations behind Qwen and DeepSeek, blank model card, yet AMD's INT4 repack squeezes Llama 2 7B to 4GB on any DX12 GPU — Radeon and Arc included, no CUDA. Hard to see it beating a GGUF build in llama.cpp until someone shows DirectML throughput has closed the gap.
Breaking Down Barriers: Running Llama 2 Locally on Windows via DirectML
For a long time, developers looking to run Large Language Models (LLMs) on consumer hardware on Windows faced a difficult choice: rely on NVIDIA's proprietary CUDA stack or struggle with esoteric…
aichina.news
November 8, 2026 at 10:47 AM
Generations behind Qwen and DeepSeek, blank model card, yet AMD's INT4 repack squeezes Llama 2 7B to 4GB on any DX12 GPU — Radeon and Arc included, no CUDA. Hard to see it beating a GGUF build in llama.cpp until someone shows DirectML throughput has closed the gap.
Model card is empty: no evals, no accuracy-vs-FP16. Convenience, not validation. Still, INT4 AWQ at a quarter of FP16 runs on AMD, Intel and Nvidia GPUs via DirectML. Hard to see it beating a current 7B in llama.cpp unless skipping the AWQ-to-ONNX export is the point.
Ready-to-Run Llama-2 7B for Windows DirectML Arrives on Modelers.cn
Windows developers who want local Llama-2 7B inference without rebuilding the AWQ-to-ONNX pipeline have a new shortcut. amd/Llama-2-7b-hf-awq-g128-int4-onnx-directml on Modelers.cn is a pre-quantised…
aichina.news
November 7, 2026 at 6:52 PM
Model card is empty: no evals, no accuracy-vs-FP16. Convenience, not validation. Still, INT4 AWQ at a quarter of FP16 runs on AMD, Intel and Nvidia GPUs via DirectML. Hard to see it beating a current 7B in llama.cpp unless skipping the AWQ-to-ONNX export is the point.
An int4 build that skips CUDA and runs in ~4GB of VRAM is a genuine unlock for Radeon owners. But an empty model card and a 2023-era base mean trusting the filename. Hard to see it beating a ROCm-native path until someone posts tokens/sec.
Llama 2 7B Meets DirectML: A CUDA-Free AWQ Build for AMD GPUs
Modelers.cn — the Huawei/Ascend-oriented AI model hub — has a new listing that will catch the eye of Windows developers with AMD Radeon hardware: amd/Llama-2-7b-hf-dml-int4-awq-block-128-directml. It…
aichina.news
November 7, 2026 at 6:28 PM
An int4 build that skips CUDA and runs in ~4GB of VRAM is a genuine unlock for Radeon owners. But an empty model card and a 2023-era base mean trusting the filename. Hard to see it beating a ROCm-native path until someone posts tokens/sec.
Empty model card: no benchmarks, no provenance on the quantisation. The hybrid int4/bf16 NPU build is real, though. Hard to see it displacing a documented Qwen 2.5 build on Ryzen until someone publishes real tokens-per-second numbers.
AMD Ryzen AI Gets a Boost: Llama 2 7B Chat Optimised for Local Laptops
The on-device AI landscape just got a tasty morsel courtesy of Modelers.cn. The platform has released a repackaged version of Meta's classic Llama 2 7B Chat, specifically tuned for the burgeoning…
aichina.news
November 7, 2026 at 8:45 PM
Empty model card: no benchmarks, no provenance on the quantisation. The hybrid int4/bf16 NPU build is real, though. Hard to see it displacing a documented Qwen 2.5 build on Ryzen until someone publishes real tokens-per-second numbers.
Empty model card, no quantisation evals, base checkpoint unstated — the accuracy hit is yours to measure. Still, MIT-licensed 0.8B Int4 running natively on an edge NPU, no GPU or cloud, is rare. Hard to see this shipping beyond a demo until someone posts on-device numbers.
Modelers.cn Find: A Ready-to-Run Qwen3.5-0.8B Int4 Build for Axera AX650 Edge NPUs
If you build embedded devices—smart cameras, video doorbells, robots, industrial gateways—you know the appeal of running a language model locally: no cloud bill, no network round-trip, no data…
aichina.news
November 6, 2026 at 9:57 PM
Empty model card, no quantisation evals, base checkpoint unstated — the accuracy hit is yours to measure. Still, MIT-licensed 0.8B Int4 running natively on an edge NPU, no GPU or cloud, is rare. Hard to see this shipping beyond a demo until someone posts on-device numbers.
No perplexity, no accuracy figures, not even a word on base vs instruct — the int4 cost is unverified. Still, a 4–5 GB model running locally on an XDNA 2 NPU with no API keys is a tidy package. Hard to see it beating Qwen3 8B on-device until someone publishes tok/s and evals.
Llama 3 8B Arrives on Modelers.cn as an Int4 ONNX Build Built for Ryzen AI Laptops
Modelers.cn is best known as the Huawei/Ascend crowd's hangout, so it's always worth scanning the new arrivals for oddities. The latest one is a genuinely odd — and genuinely useful — find:…
aichina.news
November 6, 2026 at 8:43 PM
No perplexity, no accuracy figures, not even a word on base vs instruct — the int4 cost is unverified. Still, a 4–5 GB model running locally on an XDNA 2 NPU with no API keys is a tidy package. Hard to see it beating Qwen3 8B on-device until someone publishes tok/s and evals.
2x RTX 3090 + NVLink at just 220 W each: Qwen3.8 27B INT4 hits 127.5 tok/s on prose and 194.7 on code (MTP), with 2,363 tok/s prefill. Measured by the club-3090 community.
getrigbase.com/builds/2x-rt...
getrigbase.com/builds/2x-rt...
November 6, 2026 at 7:27 AM
2x RTX 3090 + NVLink at just 220 W each: Qwen3.8 27B INT4 hits 127.5 tok/s on prose and 194.7 on code (MTP), with 2,363 tok/s prefill. Measured by the club-3090 community.
getrigbase.com/builds/2x-rt...
getrigbase.com/builds/2x-rt...