#INT8
No benchmarks, no runtime notes, CTX unexplained — the model card does very little work here. Still, 0.5B params, INT8, BSD-3-Clause, free for commercial use. Hard to see edge teams reaching for this over a stock quant export until someone publishes the accuracy delta.
A Compact Powerhouse: Qwen2.5-0.5B-Instruct-CTX-Int8 Arrives on Modelers.cn
For developers working with constrained hardware or seeking ultra-low-latency inference, the arrival of the Qwen2.5-0.5B-Instruct-CTX-Int8 model on Modelers.cn is a welcome sight. This isn't just…
aichina.news
November 20, 2026 at 4:55 PM
CUDA is the big lever. The next one after it is compute type: int8 or float16 roughly doubles throughput again on the same GPU with almost no accuracy loss. And for anyone with no NVIDIA card, faster-whisper int8 on CPU is slow but workable for short meetings.
November 19, 2026 at 10:54 PM
November 16, 2026 at 3:15 PM
⚠️ Trade-offs: int8 stays close to bf16, while 4-bit can change small details and sometimes the composition. Hands, small figures and dense small text are unreliable.
November 19, 2026 at 5:36 PM
Class exercise: an INT8 engine timed at 3.1 ms sat in a doorbell that took 47.5 ms per frame.

No bug. Preprocessing, copies, and Python NMS cost 14x the engine.

Jetson engineers: when the pipeline lags the engine, where does the time go?

#EdgeAI #Jetson
November 21, 2026 at 6:53 PM
🚀 Extras: semantic search and similar-item detection from local sentence embeddings (bge-small-en-v1.5, int8 ONNX, on CPU). Contributors' agents reach the Inbox through an MCP server. Discord mentions and an /autocraft search slash command.
November 19, 2026 at 11:01 AM
Kokoro in the browser, packed to less than 1/3 the standard size (~100MB page-load): austegard.com/experiments/...
Kokoro in the browser
Kokoro-82M text-to-speech on onnxruntime-web with every word timed: an 88 MB model (int8 vocoder, fp16 where durations and pitch are set) that matches PyTorch's word times exactly, highlighting each w...
austegard.com
November 14, 2026 at 3:42 AM
Want to generate or edit images locally on your Mac without sending prompts to a cloud API? H3 Image MLX runs MiniMax H3 Image on Apple Silicon through MLX, with a CLI and Python API. It uses W8A8 INT8 quantization with Metal acceleration, outputs 256 to 4096 pixels per side, and is open...
November 21, 2026 at 9:28 PM
ray tracing ainda é muito dependente de compute e tudo o que não é offload pro CPU (i.e. hw lumen da ue5) usam cores FP16 no PS5 e INT8 no PS5 Pro (por isso ele roda ray tracing melhor). A resolução interfere no quanto desse compute é necessário e diminui a carga mesmo
November 18, 2026 at 9:16 PM
An empty model card — no benchmarks, no params, no scripts — is thin ground for a deployment call, even with Apache-2.0 and GPTQ on the AX637. Anyone who has shipped VLMs on Axera silicon: hard to see this displacing a hand-tuned INT8 pipeline until calibration details surface.
MiniCPM-V 4.6 Goes Edge: AXERA’s GPTQ Build Targets AX637 NPUs
Modelers.cn has a new listing that will interest developers working close to the metal: AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637. In plain terms, it is a GPTQ-quantised build of MiniCPM-V 4.6 — a…
aichina.news
November 21, 2026 at 7:19 PM
脳 NPU 、10 TOPS (INT8)
November 12, 2026 at 3:20 AM
The authors compress the gemma-3-12b-it model using int8 weights or fp8 activations without resolved loss, while int4 quantization leaves a small residual.

Source: arXiv cs.LG
Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality
Abstract page for arXiv paper 2611.18587: Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality
arxiv.org
November 17, 2026 at 9:58 AM
An 8-bit Qwen2.5 1.5B under BSD-3-Clause is a cheap starting point for constrained boxes, but the card ships no benchmarks, no Ascend guidance and no sign of quantisation loss. Hard to see this beating a well-tested llama.cpp int4 build until the publisher documents what it actually runs on.
A Tiny Qwen for Tight Spaces: AXERA-TECH's 1.5B GPTQ Int8 Model Arrives on Modelers.cn
Modelers.cn, the Huawei/Ascend AI ecosystem platform, has listed a new small-model option that deserves a look from developers working under memory and compute constraints:…
aichina.news
November 20, 2026 at 1:45 PM
Huh that's interesting, trained new vision models for work based on this week's data, and I had a regression. Not on the trained model, but on the int8 quant that runs on a Pi. Not seen that before.
November 15, 2026 at 11:33 AM
>続いて試すのはComfyUIだ。(中略)「FLUX.2 [dev]」はFP8混合精度の生成モデルで1,024×1,024ドットの画像を1枚生成。「MiniMax H3」はINT8版を使い、約5秒の動画を生成した。ステップ数はどちらも20だ。(中略)画像は1枚当たり約21秒、動画は1回当たり約52秒。

128GBのユニファイドメモリを半分割り当てて AI の画像や映像が1分以内で生成できるのは魅力的。(自分のよわよわ環境だとMiniMax H3が30分とか出てた)
ただ100万超えかぁ・・・。(ASUSの小型PCは64GBだがこれでも81万円) […]
Original post on kemonodon.club
kemonodon.club
November 20, 2026 at 2:45 PM
From 507 MB and C++ dependencies to this 7-line loop: how Modern Web Guidance cut query embed time to 30 µs → goo.gle/4zkFwWr

for (const [i, f] of enc) {
const wt = scales[i] * f;
const offset = i * 256;
for (let k = 0; k < 256; k++) {
out[k] += wt * table[offset + k];
}
}
November 12, 2026 at 7:33 PM
The story here is really the packing process; a bunch of these models have a lot of excess precision where it doesn’t matter a whole lot. In this case: 4-bit token tables and int8 weights, no measurable reduction in scores. See details above.
November 11, 2026 at 1:43 PM
Serving an LLM in production usually means juggling quantization, batching and an API layer that your apps can actually call. LMDeploy bundles all of it: TurboMind and PyTorch engines, continuous batching, AWQ/GPTQ and INT4/INT8 KV cache quantization, plus an OpenAI-compatible server. It's...
November 11, 2026 at 3:02 PM
えどもの、PvPの可能性あるって書いてたっけ?(INT8の作者)
November 10, 2026 at 1:28 AM
This rescoring approach is very cheap, and heavily negates the int8 performance hit. After that, we just need to load the actual images, and we can send that to the user.

For example, vs exact fp32 search:
int8: 97.24% recall@100
int8 + fp16/fp32 rescoring: 99.94% recall@100
November 9, 2026 at 4:27 PM
Nice compression numbers if they hold up under real eval, not just demo clips. INT8 + 6 steps is the part I'd want benchmarked hard.

https://github.com/Video-Rebirth/hyperflow-h3-flash
November 11, 2026 at 3:26 PM
smolmaxxing prétend faire tourner des LLM sur des appareils avec seulement quelques centaines de mégaoctets de RAM, en misant sur la quantisation int8/int4 pour réduire la mémoire requise.

techfi24.com #LLM
Smolmaxxing permet d'exécuter des LLMs sur appareils anciens
Le dépôt GitHub smolmaxxing se décrit comme une tentative de faire fonctionner des modèles de langage de grande taille (LLM) sur des appareils à faible puissance et anciens. Aucun détail technique sup
www.techfi24.com
November 12, 2026 at 5:10 PM
6 steps is the wild part – diffusion models usually need 20-50 for decent output. Did they distill the step count down separately from the INT8 quantization, or is it one combined pass? Order of operations matters a lot for where quality loss creeps in.
November 11, 2026 at 3:31 PM