machine learning needs a nonlinear function, such as a transition between a float32 layer and an INT4 layer
March 14, 2026 at 1:07 AM
machine learning needs a nonlinear function, such as a transition between a float32 layer and an INT4 layer
Alibaba Ant's Ling-3.0-tiny is now available as open-weight:
BF16: huggingface.co/inclusionAI/...
FP8: huggingface.co/inclusionAI/...
INT4: huggingface.co/inclusionAI/...
BF16: huggingface.co/inclusionAI/...
FP8: huggingface.co/inclusionAI/...
INT4: huggingface.co/inclusionAI/...
August 10, 2026 at 5:24 PM
Alibaba Ant's Ling-3.0-tiny is now available as open-weight:
BF16: huggingface.co/inclusionAI/...
FP8: huggingface.co/inclusionAI/...
INT4: huggingface.co/inclusionAI/...
BF16: huggingface.co/inclusionAI/...
FP8: huggingface.co/inclusionAI/...
INT4: huggingface.co/inclusionAI/...
Gemma 27b runs really well in int4 and has that big model feel
April 28, 2025 at 12:44 AM
Gemma 27b runs really well in int4 and has that big model feel
Qwen3.5 GPTQ int4 native quants are available
yeah, that means you can run 35B-A3B at 1M context in under 32 GB of memory (including activations!)
huggingface.co/collections/...
yeah, that means you can run 35B-A3B at 1M context in under 32 GB of memory (including activations!)
huggingface.co/collections/...
Qwen3.5 - a Qwen Collection
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
March 4, 2026 at 4:05 PM
Qwen3.5 GPTQ int4 native quants are available
yeah, that means you can run 35B-A3B at 1M context in under 32 GB of memory (including activations!)
huggingface.co/collections/...
yeah, that means you can run 35B-A3B at 1M context in under 32 GB of memory (including activations!)
huggingface.co/collections/...
OpenBMB's MiniCPM5-1B
INT4 weights = just ~0.5GB — runs on phones, browsers, laptops.
🤗 Hugging Face: huggingface.openbmb.com/model/openbm...
💻 GitHub: github.com/OpenBMB/Mini...
🔭 Modelscope: modelscope.cn/models/OpenB...
INT4 weights = just ~0.5GB — runs on phones, browsers, laptops.
🤗 Hugging Face: huggingface.openbmb.com/model/openbm...
💻 GitHub: github.com/OpenBMB/Mini...
🔭 Modelscope: modelscope.cn/models/OpenB...
May 30, 2026 at 5:16 PM
OpenBMB's MiniCPM5-1B
INT4 weights = just ~0.5GB — runs on phones, browsers, laptops.
🤗 Hugging Face: huggingface.openbmb.com/model/openbm...
💻 GitHub: github.com/OpenBMB/Mini...
🔭 Modelscope: modelscope.cn/models/OpenB...
INT4 weights = just ~0.5GB — runs on phones, browsers, laptops.
🤗 Hugging Face: huggingface.openbmb.com/model/openbm...
💻 GitHub: github.com/OpenBMB/Mini...
🔭 Modelscope: modelscope.cn/models/OpenB...
fp8をハードウェア的に対応できなくても別に問題ない
int4/int8はちゃんと計算できるし
int4/int8はちゃんと計算できるし
September 22, 2026 at 6:20 AM
fp8をハードウェア的に対応できなくても別に問題ない
int4/int8はちゃんと計算できるし
int4/int8はちゃんと計算できるし
Run Gemma 3 27B on your desktop GPU 🔥
Our new QAT-optimized int4 models slash VRAM needs (54GB -> 14.1GB) while maintaining quality.
Now accessible on consumer cards like the NVIDIA RTX 3090 via ollama, hugging face, lmstudio, kaggle and llama.cpp
developers.googleblog.com/en/gemma-3-q...
Our new QAT-optimized int4 models slash VRAM needs (54GB -> 14.1GB) while maintaining quality.
Now accessible on consumer cards like the NVIDIA RTX 3090 via ollama, hugging face, lmstudio, kaggle and llama.cpp
developers.googleblog.com/en/gemma-3-q...
April 18, 2025 at 4:09 PM
Run Gemma 3 27B on your desktop GPU 🔥
Our new QAT-optimized int4 models slash VRAM needs (54GB -> 14.1GB) while maintaining quality.
Now accessible on consumer cards like the NVIDIA RTX 3090 via ollama, hugging face, lmstudio, kaggle and llama.cpp
developers.googleblog.com/en/gemma-3-q...
Our new QAT-optimized int4 models slash VRAM needs (54GB -> 14.1GB) while maintaining quality.
Now accessible on consumer cards like the NVIDIA RTX 3090 via ollama, hugging face, lmstudio, kaggle and llama.cpp
developers.googleblog.com/en/gemma-3-q...
Moonshot AI's Kimi-K2-Thinking's INT4 QAT explanation
"INT4 QAT is weight-only with fake-quantization: we keep the original BF16 weights in memory, during the forward pass we on-the-fly quantize them to INT4 and immediately de-quantize back to BF16 for the actual computation.
"INT4 QAT is weight-only with fake-quantization: we keep the original BF16 weights in memory, during the forward pass we on-the-fly quantize them to INT4 and immediately de-quantize back to BF16 for the actual computation.
November 10, 2025 at 11:38 PM
Moonshot AI's Kimi-K2-Thinking's INT4 QAT explanation
"INT4 QAT is weight-only with fake-quantization: we keep the original BF16 weights in memory, during the forward pass we on-the-fly quantize them to INT4 and immediately de-quantize back to BF16 for the actual computation.
"INT4 QAT is weight-only with fake-quantization: we keep the original BF16 weights in memory, during the forward pass we on-the-fly quantize them to INT4 and immediately de-quantize back to BF16 for the actual computation.
Int4 flops :P
November 30, 2024 at 3:40 AM
Int4 flops :P
HY-1.8B-2Bit 🔥 A 2-bit on-device LLM released by Tencent
huggingface.co/AngelSlim/HY...
huggingface.co/AngelSlim/HY...
✨ GGUF available
✨ Near INT4 performance at just 2-bit, with full reasoning preserved
huggingface.co/AngelSlim/HY...
huggingface.co/AngelSlim/HY...
✨ GGUF available
✨ Near INT4 performance at just 2-bit, with full reasoning preserved
AngelSlim/HY-1.8B-2Bit-GGUF · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
February 10, 2026 at 8:54 AM
HY-1.8B-2Bit 🔥 A 2-bit on-device LLM released by Tencent
huggingface.co/AngelSlim/HY...
huggingface.co/AngelSlim/HY...
✨ GGUF available
✨ Near INT4 performance at just 2-bit, with full reasoning preserved
huggingface.co/AngelSlim/HY...
huggingface.co/AngelSlim/HY...
✨ GGUF available
✨ Near INT4 performance at just 2-bit, with full reasoning preserved
It’s a bit funny that even FAIR at Meta doesn’t evaluate Llama 4.
They present a new 4-bit weight quantization solution for large language models (LLMs), called any4, which yields higher accuracy compared to other related 4-bit numeric representation types: int4, fp4 and nf4.
They present a new 4-bit weight quantization solution for large language models (LLMs), called any4, which yields higher accuracy compared to other related 4-bit numeric representation types: int4, fp4 and nf4.
July 9, 2025 at 2:34 AM
It’s a bit funny that even FAIR at Meta doesn’t evaluate Llama 4.
They present a new 4-bit weight quantization solution for large language models (LLMs), called any4, which yields higher accuracy compared to other related 4-bit numeric representation types: int4, fp4 and nf4.
They present a new 4-bit weight quantization solution for large language models (LLMs), called any4, which yields higher accuracy compared to other related 4-bit numeric representation types: int4, fp4 and nf4.
sigam minha banda galera
Int4: mefuriaband
Tiktok: mefuriaband
www.instagram.com/mefuriaband?...
#sdv #banda #band
Int4: mefuriaband
Tiktok: mefuriaband
www.instagram.com/mefuriaband?...
#sdv #banda #band
September 8, 2024 at 7:48 PM
sigam minha banda galera
Int4: mefuriaband
Tiktok: mefuriaband
www.instagram.com/mefuriaband?...
#sdv #banda #band
Int4: mefuriaband
Tiktok: mefuriaband
www.instagram.com/mefuriaband?...
#sdv #banda #band
use the official GPTQ-int4 from huggingface, unless you REALLY want to haul bf16
March 7, 2026 at 10:30 PM
use the official GPTQ-int4 from huggingface, unless you REALLY want to haul bf16
LLMのINT8・INT4量子化を実装条件から比較する | kas's blog #zenn
https://zenn.dev/kas_blog/articles/20260509-llm-18-quantization
https://zenn.dev/kas_blog/articles/20260509-llm-18-quantization
LLMのINT8・INT4量子化を実装条件から比較する
zenn.dev
September 21, 2026 at 9:15 PM
LLMのINT8・INT4量子化を実装条件から比較する | kas's blog #zenn
https://zenn.dev/kas_blog/articles/20260509-llm-18-quantization
https://zenn.dev/kas_blog/articles/20260509-llm-18-quantization
昨日に続いてQwen3.8-Flash-Nextで遊んでみてるんだけど、さすがにINT4とはいえ177Bもあるモデルだから知識もちゃんとしてるなぁ
周辺情報も全部合ってるのはなかなかない
周辺情報も全部合ってるのはなかなかない
September 19, 2026 at 4:21 AM
昨日に続いてQwen3.8-Flash-Nextで遊んでみてるんだけど、さすがにINT4とはいえ177Bもあるモデルだから知識もちゃんとしてるなぁ
周辺情報も全部合ってるのはなかなかない
周辺情報も全部合ってるのはなかなかない
GPT/3.5 (compatible) Gemini-Pro/1.5 (PaLM-2; Bard) Claude/3-sonnet LLaMA/3.2-3B-int4 (like LLaMA/3-8B) Gemma-7B (SearchGrounded/2.0; like Flash)
July 1, 2025 at 10:26 AM
GPT/3.5 (compatible) Gemini-Pro/1.5 (PaLM-2; Bard) Claude/3-sonnet LLaMA/3.2-3B-int4 (like LLaMA/3-8B) Gemma-7B (SearchGrounded/2.0; like Flash)
Moonshot just released the thinking version of their K2 model. One big change is that the experts (except the shared expert) are quantized to INT4. The #1 question I have on it now is whether the reasoning training has solved its frequent hallucination. moonshotai.github.io/Kimi-K2/thin...
November 6, 2025 at 3:39 PM
Moonshot just released the thinking version of their K2 model. One big change is that the experts (except the shared expert) are quantized to INT4. The #1 question I have on it now is whether the reasoning training has solved its frequent hallucination. moonshotai.github.io/Kimi-K2/thin...
Takuto Ando, Yasuhiko Nakashima: Implementation and Evaluation of BitNet Inference on a CGLA by Signed-Int4 Instructions https://arxiv.org/abs/2609.27453 https://arxiv.org/pdf/2609.27453 https://arxiv.org/html/2609.27453
September 24, 2026 at 6:38 AM
Takuto Ando, Yasuhiko Nakashima: Implementation and Evaluation of BitNet Inference on a CGLA by Signed-Int4 Instructions https://arxiv.org/abs/2609.27453 https://arxiv.org/pdf/2609.27453 https://arxiv.org/html/2609.27453
The model is built for fine-tuning on focused tasks where efficiency is everything:
🔹 Massive 256k vocabulary for expert fine-tuning
🔹 Extreme energy efficiency for on-device AI
🔹 Production-ready with INT4 Quantization Aware Training
🔹 Massive 256k vocabulary for expert fine-tuning
🔹 Extreme energy efficiency for on-device AI
🔹 Production-ready with INT4 Quantization Aware Training
August 14, 2025 at 8:28 PM
The model is built for fine-tuning on focused tasks where efficiency is everything:
🔹 Massive 256k vocabulary for expert fine-tuning
🔹 Extreme energy efficiency for on-device AI
🔹 Production-ready with INT4 Quantization Aware Training
🔹 Massive 256k vocabulary for expert fine-tuning
🔹 Extreme energy efficiency for on-device AI
🔹 Production-ready with INT4 Quantization Aware Training
now downloading stepfun-ai/Step-3.5-Flash-Int4
February 2, 2026 at 3:24 PM
now downloading stepfun-ai/Step-3.5-Flash-Int4
素晴らしいLoRAを見つけた。int4 turbo convert 600x856の4 seconds生成でこの質感はええぞ。キャラクターはオリジナルちゃん。
September 22, 2026 at 4:55 PM
素晴らしいLoRAを見つけた。int4 turbo convert 600x856の4 seconds生成でこの質感はええぞ。キャラクターはオリジナルちゃん。
Sage Attention the next Flash Attention? 🤔
> 3x speed up over Flash Attention2, maintaining 99% performance
> INT4/8 for Q and K matrices, and FP8/16 for P and V + smoothing methods for Q and V
> Drop-in replacement of torch scaled_dot_product_attention
> SageAttention 2 code to be released soon
> 3x speed up over Flash Attention2, maintaining 99% performance
> INT4/8 for Q and K matrices, and FP8/16 for P and V + smoothing methods for Q and V
> Drop-in replacement of torch scaled_dot_product_attention
> SageAttention 2 code to be released soon
November 20, 2024 at 8:11 AM
Sage Attention the next Flash Attention? 🤔
> 3x speed up over Flash Attention2, maintaining 99% performance
> INT4/8 for Q and K matrices, and FP8/16 for P and V + smoothing methods for Q and V
> Drop-in replacement of torch scaled_dot_product_attention
> SageAttention 2 code to be released soon
> 3x speed up over Flash Attention2, maintaining 99% performance
> INT4/8 for Q and K matrices, and FP8/16 for P and V + smoothing methods for Q and V
> Drop-in replacement of torch scaled_dot_product_attention
> SageAttention 2 code to be released soon
github.com/int4-org/Com... << ...Commonly usable Java classes without dependencies... >> #datastructures #java #utilities
GitHub - int4-org/Common: Commonly usable Java classes without dependencies.
Commonly usable Java classes without dependencies. - int4-org/Common
github.com
April 13, 2026 at 8:15 PM
github.com/int4-org/Com... << ...Commonly usable Java classes without dependencies... >> #datastructures #java #utilities
Please consider using the Unsloth quants instead. Also not that int4 has worse precision than MXFP4 if you are on AMD or Mac use these quants instead. Also, test and tweak your hyperparameters (tempatur, topP, repetition penalty, topK).
These change a lot!
These change a lot!
use the official GPTQ-int4 from huggingface, unless you REALLY want to haul bf16
March 8, 2026 at 3:51 AM
Please consider using the Unsloth quants instead. Also not that int4 has worse precision than MXFP4 if you are on AMD or Mac use these quants instead. Also, test and tweak your hyperparameters (tempatur, topP, repetition penalty, topK).
These change a lot!
These change a lot!