argon how are you still finding ways to optimize llamacpp it’s been like 1h30
October 31, 2026 at 12:00 AM
argon how are you still finding ways to optimize llamacpp it’s been like 1h30
nice! I really gotta sic an agent on my tweaked llamacpp again, I think we're still leaving significant perf on the table
October 31, 2026 at 1:05 AM
nice! I really gotta sic an agent on my tweaked llamacpp again, I think we're still leaving significant perf on the table
InferSSD Runs a 763-Billion-Parameter DeepSeek Model on a 48 GB MacBook Pro by Reading Experts From the SSD
lockergnome.com/inferssd-runs-763b-deep…
#LocalAi #LlamaCpp #deepseek #OpenSource
lockergnome.com/inferssd-runs-763b-deep…
#LocalAi #LlamaCpp #deepseek #OpenSource
October 31, 2026 at 7:20 PM
InferSSD Runs a 763-Billion-Parameter DeepSeek Model on a 48 GB MacBook Pro by Reading Experts From the SSD
lockergnome.com/inferssd-runs-763b-deep…
#LocalAi #LlamaCpp #deepseek #OpenSource
lockergnome.com/inferssd-runs-763b-deep…
#LocalAi #LlamaCpp #deepseek #OpenSource
Here'a cool experiment you can test for free:
Download llama cpp
Download a 4B model (Q4 quantization)
Run llamacpp with that model
See the amount of RAM and CPU effort required by llamacpp
Now consider that a commercial (paid) model is on 800B-3000B range, and imagine the hardware required.
Download llama cpp
Download a 4B model (Q4 quantization)
Run llamacpp with that model
See the amount of RAM and CPU effort required by llamacpp
Now consider that a commercial (paid) model is on 800B-3000B range, and imagine the hardware required.
Breaking news: Alphabet shares tumbled 13% on Thursday after the Google owner revealed that 'AI mode' searches were threatening profits at its core advertising business. ft.trib.al/g8Q9IPx
October 29, 2026 at 3:46 PM
Here'a cool experiment you can test for free:
Download llama cpp
Download a 4B model (Q4 quantization)
Run llamacpp with that model
See the amount of RAM and CPU effort required by llamacpp
Now consider that a commercial (paid) model is on 800B-3000B range, and imagine the hardware required.
Download llama cpp
Download a 4B model (Q4 quantization)
Run llamacpp with that model
See the amount of RAM and CPU effort required by llamacpp
Now consider that a commercial (paid) model is on 800B-3000B range, and imagine the hardware required.
huggingface.co/GrootTeam/Gro…
ちょうどいいサイズだなあと思ったらllamacppでは扱えないっぽくてかなしみ
Original: https://x.com/4ba_ba_baba/status/2117758101596278913
(リンク: https://huggingface.co/GrootTeam/Groot-19B-A3B-261023)
ちょうどいいサイズだなあと思ったらllamacppでは扱えないっぽくてかなしみ
Original: https://x.com/4ba_ba_baba/status/2117758101596278913
(リンク: https://huggingface.co/GrootTeam/Groot-19B-A3B-261023)
November 3, 2026 at 11:37 PM
huggingface.co/GrootTeam/Gro…
ちょうどいいサイズだなあと思ったらllamacppでは扱えないっぽくてかなしみ
Original: https://x.com/4ba_ba_baba/status/2117758101596278913
(リンク: https://huggingface.co/GrootTeam/Groot-19B-A3B-261023)
ちょうどいいサイズだなあと思ったらllamacppでは扱えないっぽくてかなしみ
Original: https://x.com/4ba_ba_baba/status/2117758101596278913
(リンク: https://huggingface.co/GrootTeam/Groot-19B-A3B-261023)
Una RX 580 de 8 GB ejecutando un modelo de 30B a 21 tokens/s. Medí cinco modelos en llama.cpp con Vulkan. Resultados, memoria compartida y límites reales.
#IALocal #RX580 #LlamaCpp #Homelab #OpenSource
#IALocal #RX580 #LlamaCpp #Homelab #OpenSource
October 29, 2026 at 12:31 PM
Una RX 580 de 8 GB ejecutando un modelo de 30B a 21 tokens/s. Medí cinco modelos en llama.cpp con Vulkan. Resultados, memoria compartida y límites reales.
#IALocal #RX580 #LlamaCpp #Homelab #OpenSource
#IALocal #RX580 #LlamaCpp #Homelab #OpenSource
InferSSD Runs a 763-Billion-Parameter DeepSeek Model on a 48 GB MacBook Pro by Reading Experts From the SSD
https://lockergnome.com/inferssd-runs-763b-deepseek-on-48gb-macbook-by-streaming-experts-from-ssd/
#localai #llamacpp #deepseek #opensource
https://lockergnome.com/inferssd-runs-763b-deepseek-on-48gb-macbook-by-streaming-experts-from-ssd/
#localai #llamacpp #deepseek #opensource
November 1, 2026 at 1:21 AM
InferSSD Runs a 763-Billion-Parameter DeepSeek Model on a 48 GB MacBook Pro by Reading Experts From the SSD
https://lockergnome.com/inferssd-runs-763b-deepseek-on-48gb-macbook-by-streaming-experts-from-ssd/
#localai #llamacpp #deepseek #opensource
https://lockergnome.com/inferssd-runs-763b-deepseek-on-48gb-macbook-by-streaming-experts-from-ssd/
#localai #llamacpp #deepseek #opensource
Gaijin Entertainment launched daslang, a scripting language featuring a compiler and LLVM-based JIT that integrated LLM, ASR, and TTS into dasllama. Benchmarks demonstrated a 5x speedup in speech recognition compared to llama.cpp on CPU.
#Daslang #LlamaCPP #SpeechTOText
#Daslang #LlamaCPP #SpeechTOText
dasllama runs speech-to-text 5x faster than llama.cpp on CPU
Source: Lobsters
realfeedapp.com
November 1, 2026 at 7:20 AM
Gaijin Entertainment launched daslang, a scripting language featuring a compiler and LLVM-based JIT that integrated LLM, ASR, and TTS into dasllama. Benchmarks demonstrated a 5x speedup in speech recognition compared to llama.cpp on CPU.
#Daslang #LlamaCPP #SpeechTOText
#Daslang #LlamaCPP #SpeechTOText
the engines like llamacpp are designed to run on anything with most models. theres also the fact that people running locally are doing stuff that actually end up somewhat hurt model performance for pretty big gains like quantizing kv cache or boosting token scores that are in cached mem
October 28, 2026 at 4:16 AM
the engines like llamacpp are designed to run on anything with most models. theres also the fact that people running locally are doing stuff that actually end up somewhat hurt model performance for pretty big gains like quantizing kv cache or boosting token scores that are in cached mem
LFM3をllama.cppで手元の会話の補助に。速さは足りたが、専門用語の聞き取りでWhisperに届かなかった | maruhana #zenn
https://zenn.dev/maruhana/articles/lfm3-audio-llamacpp-conversation-aid
https://zenn.dev/maruhana/articles/lfm3-audio-llamacpp-conversation-aid
LFM3をllama.cppで手元の会話の補助に。速さは足りたが、専門用語の聞き取りでWhisperに届かなかった
zenn.dev
November 4, 2026 at 6:53 PM
LFM3をllama.cppで手元の会話の補助に。速さは足りたが、専門用語の聞き取りでWhisperに届かなかった | maruhana #zenn
https://zenn.dev/maruhana/articles/lfm3-audio-llamacpp-conversation-aid
https://zenn.dev/maruhana/articles/lfm3-audio-llamacpp-conversation-aid
I should note that llamacpp is running an offline gemma 4 e4b LLM, so not entirely de-googled, but a big step towards it.
October 29, 2026 at 3:47 AM
I should note that llamacpp is running an offline gemma 4 e4b LLM, so not entirely de-googled, but a big step towards it.
✍️ New blog post by xbill
Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4
#gemma #llamacpp #aws #quantization
Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4
#gemma #llamacpp #aws #quantization
Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4
Gemma 4 E2B from BF16 to Q2_K, speed against GSM8K accuracy, on an AWS L4 with byte-identical GGUF files and the same llama.cpp build as the Tesla T4. The L4 decodes 1.25 to 1.89 times faster, Q5_K_M is still the best trade, and switching GPUs changes nearly as many grades as quantizing from BF16 to Q5_K_M.
dev.to
October 29, 2026 at 2:45 PM
✍️ New blog post by xbill
Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4
#gemma #llamacpp #aws #quantization
Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4
#gemma #llamacpp #aws #quantization
💫 Руководство по запуску локальной нейросетевой модели llama-cpp с собственным веб-интерфейсом.
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
October 28, 2026 at 10:00 AM
💫 Руководство по запуску локальной нейросетевой модели llama-cpp с собственным веб-интерфейсом.
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
💫 Руководство по запуску локальной нейросетевой модели llama-cpp с собственным веб-интерфейсом.
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
October 28, 2026 at 10:00 AM
💫 Руководство по запуску локальной нейросетевой модели llama-cpp с собственным веб-интерфейсом.
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
• скачивание и сборка llama-cpp
• загрузка тестовой модели
• настройка параметров mmap и памяти
• запуск сервера по адресу 127.0.0.1
#LlamaCPP #ЛокальныйИИ #ВЕБИнтерфейс
🌐 ТГ / ВК
🔗 https://habr.com/…
Exciting news! With a combination of a llamacpp container running on my home server, with an old RTX2080, #homeassistant, whisper, piper, and some M5 Atom Echos running #esphome I finally have a path to a fully de-googled home automation voice control system. I have a Respeaker XVF3800 on the […]
Original post on howse.social
howse.social
October 29, 2026 at 3:40 AM
Exciting news! With a combination of a llamacpp container running on my home server, with an old RTX2080, #homeassistant, whisper, piper, and some M5 Atom Echos running #esphome I finally have a path to a fully de-googled home automation voice control system. I have a Respeaker XVF3800 on the […]
same experience here
I would add: use llama-swap github.com/mostlygeek/l...
to load / unload / run multiple models on demand
the recipes in the article can easily be ported to llama-swap an it supports any backend (i use it with both llamacpp/vLLM/sgLang)
I would add: use llama-swap github.com/mostlygeek/l...
to load / unload / run multiple models on demand
the recipes in the article can easily be ported to llama-swap an it supports any backend (i use it with both llamacpp/vLLM/sgLang)
GitHub - mostlygeek/llama-swap: Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc
Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc - mostlygeek/llama-swap
github.com
October 28, 2026 at 1:03 PM
same experience here
I would add: use llama-swap github.com/mostlygeek/l...
to load / unload / run multiple models on demand
the recipes in the article can easily be ported to llama-swap an it supports any backend (i use it with both llamacpp/vLLM/sgLang)
I would add: use llama-swap github.com/mostlygeek/l...
to load / unload / run multiple models on demand
the recipes in the article can easily be ported to llama-swap an it supports any backend (i use it with both llamacpp/vLLM/sgLang)
Claude Opus 5.5 asiste la b11940 de llama.cpp
¿Qué suma la b11940 de llama.cpp? Claude Opus 5.5 asistió el PR de deferred tools que ya corre en master desde el 27 de octubre de 2026, según GitHub
#llamacpp #claudeopus55 #deferredtools #opensource #github
¿Qué suma la b11940 de llama.cpp? Claude Opus 5.5 asistió el PR de deferred tools que ya corre en master desde el 27 de octubre de 2026, según GitHub
#llamacpp #claudeopus55 #deferredtools #opensource #github
Claude Opus 5.5 asiste la b11940 de llama.cpp
La b11940 de llama.cpp suma deferred tools y tool_reference content parts del PR #31256, asistido por Claude Opus 5.5. Mergeado a master el 27 de octubre de 2026.
blog.donweb.com
October 28, 2026 at 7:33 AM
Claude Opus 5.5 asiste la b11940 de llama.cpp
¿Qué suma la b11940 de llama.cpp? Claude Opus 5.5 asistió el PR de deferred tools que ya corre en master desde el 27 de octubre de 2026, según GitHub
#llamacpp #claudeopus55 #deferredtools #opensource #github
¿Qué suma la b11940 de llama.cpp? Claude Opus 5.5 asistió el PR de deferred tools que ya corre en master desde el 27 de octubre de 2026, según GitHub
#llamacpp #claudeopus55 #deferredtools #opensource #github
CVE-2026-118375 - llama.cpp
Versions of llama.cpp between the indicated revisions have a flaw in the network code that lets an unauthenticated remote system send special data and overwrite memory on the…
Too many irrelevant or confusing CVEs? Use stackflag.com
#llamacpp #ggmlorg #CVE #infosec
Versions of llama.cpp between the indicated revisions have a flaw in the network code that lets an unauthenticated remote system send special data and overwrite memory on the…
Too many irrelevant or confusing CVEs? Use stackflag.com
#llamacpp #ggmlorg #CVE #infosec
CVE-2026-118375: llama.cpp can be crashed or hijacked by remote users
Versions of llama.cpp between the indicated revisions have a flaw in the network code that lets an unauthenticated remote system send special data and.
stackflag.com
October 28, 2026 at 7:10 AM
CVE-2026-118375 - llama.cpp
Versions of llama.cpp between the indicated revisions have a flaw in the network code that lets an unauthenticated remote system send special data and overwrite memory on the…
Too many irrelevant or confusing CVEs? Use stackflag.com
#llamacpp #ggmlorg #CVE #infosec
Versions of llama.cpp between the indicated revisions have a flaw in the network code that lets an unauthenticated remote system send special data and overwrite memory on the…
Too many irrelevant or confusing CVEs? Use stackflag.com
#llamacpp #ggmlorg #CVE #infosec
✍️ New blog post by xbill
Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs
#gemma #llamacpp #aws #cuda
Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs
#gemma #llamacpp #aws #cuda
Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs
A 24 GB NVIDIA L4 serves Gemma 4 E2B and E4B at only 1.2-1.3x an 8 GB desktop RTX 3050 for one user, with the same answers, because a 4-bit small model uses 13-29% of the card. Pick the model for the hardware, and the hardware for the model.
dev.to
October 27, 2026 at 1:37 PM
🚀 llama.cpp b11871 and b11873: JSON Depth Limit and Q2_K on Adreno
Two llama.cpp releases dated 2026-10-25 change different areas.
Inside:
• Endpoints That Answer 400 Past 128 Levels
#LlamaCpp #LLMInference
Full breakdown, every claim linked to its source 👇
Two llama.cpp releases dated 2026-10-25 change different areas.
Inside:
• Endpoints That Answer 400 Past 128 Levels
#LlamaCpp #LLMInference
Full breakdown, every claim linked to its source 👇
llama.cpp b11871 and b11873: JSON Depth Limit and Q2_K on Adreno
llama.cpp b11871 returns HTTP 400 for JSON nested past 128 levels, and b11873 adds initial Q2_K support to the OpenCL backend for Adreno GPUs.
thestacksentinel.com
October 25, 2026 at 10:26 AM
🚀 llama.cpp b11871 and b11873: JSON Depth Limit and Q2_K on Adreno
Two llama.cpp releases dated 2026-10-25 change different areas.
Inside:
• Endpoints That Answer 400 Past 128 Levels
#LlamaCpp #LLMInference
Full breakdown, every claim linked to its source 👇
Two llama.cpp releases dated 2026-10-25 change different areas.
Inside:
• Endpoints That Answer 400 Past 128 Levels
#LlamaCpp #LLMInference
Full breakdown, every claim linked to its source 👇
If you're running local models, do you prefer using a high-level wrapper like Ollama or LM Studio, or do you prefer going straight to llama.cpp to have full control over batch size, GPU layers, and speculative decoding?
#gpuoptimization #batchinference #selfhosting #llm #localai #llamacpp
#gpuoptimization #batchinference #selfhosting #llm #localai #llamacpp
October 21, 2026 at 11:50 PM
If you're running local models, do you prefer using a high-level wrapper like Ollama or LM Studio, or do you prefer going straight to llama.cpp to have full control over batch size, GPU layers, and speculative decoding?
#gpuoptimization #batchinference #selfhosting #llm #localai #llamacpp
#gpuoptimization #batchinference #selfhosting #llm #localai #llamacpp
yzma 1.30 is over 2.6 times faster at embeddings than Ollama or @docker.com Model Runner!
New benchmarks here:
yzma.ai/docs/referen...
#llamacpp #golang #localAI #ollama #dmr
New benchmarks here:
yzma.ai/docs/referen...
#llamacpp #golang #localAI #ollama #dmr
Benchmarks
How fast yzma is on many machines.
yzma.ai
October 19, 2026 at 2:12 PM
yzma 1.30 is over 2.6 times faster at embeddings than Ollama or @docker.com Model Runner!
New benchmarks here:
yzma.ai/docs/referen...
#llamacpp #golang #localAI #ollama #dmr
New benchmarks here:
yzma.ai/docs/referen...
#llamacpp #golang #localAI #ollama #dmr
another thing gemma 4 is bad at is just working properly at all; the chat template issues that persisted for far longer than they should have, 12B segfault with llamacpp for weeks (non-reproducible by friends so i couldn't file an issue with llamacpp), y'all doing gemma.cpp work in a dev branch
October 18, 2026 at 11:34 PM
another thing gemma 4 is bad at is just working properly at all; the chat template issues that persisted for far longer than they should have, 12B segfault with llamacpp for weeks (non-reproducible by friends so i couldn't file an issue with llamacpp), y'all doing gemma.cpp work in a dev branch
vLLM, SGLang, LMDeploy and llama.cpp differ in more than speed.
www.techiesjournal.com/choosing-ai-...
#vLLM #SGLang #LMDeploy #llamacpp #AIInference #LLMOps #MachineLearning #MLOps #GPUComputing #DeepLearning #ArtificialIntelligence #TechiesJournal #Nvidia
www.techiesjournal.com/choosing-ai-...
#vLLM #SGLang #LMDeploy #llamacpp #AIInference #LLMOps #MachineLearning #MLOps #GPUComputing #DeepLearning #ArtificialIntelligence #TechiesJournal #Nvidia
Choosing an AI Inference Engine for Production: What to Compare
How to choose an AI inference engine for production: compatibility, latency metrics, benchmark pitfalls, and where vLLM, SGLang, LMDeploy and llama.cpp fit.
www.techiesjournal.com
October 21, 2026 at 4:19 AM
vLLM, SGLang, LMDeploy and llama.cpp differ in more than speed.
www.techiesjournal.com/choosing-ai-...
#vLLM #SGLang #LMDeploy #llamacpp #AIInference #LLMOps #MachineLearning #MLOps #GPUComputing #DeepLearning #ArtificialIntelligence #TechiesJournal #Nvidia
www.techiesjournal.com/choosing-ai-...
#vLLM #SGLang #LMDeploy #llamacpp #AIInference #LLMOps #MachineLearning #MLOps #GPUComputing #DeepLearning #ArtificialIntelligence #TechiesJournal #Nvidia