The clarification you posted is extremely useful — most people assume Nemotron‑3‑Omni can be enabled with a “standard mmproj”, but Omni is not a symmetric multimodal model. Vision, audio and video each have their own encoder geometry, and llama.cpp upstream only implements the vision path. That’s why the usual GGUFs appear to “work” but silently drop audio/video. Your unified mmproj solves the real problem: it carries all three towers in one file (C‑RADIO, full Parakeet/FastConformer, and the temporal video embedder). And the fact that the video graph matches NVIDIA’s PyTorch reference to 0.0019% relative L2 is a strong indicator that the conversion is correct. Below is a small diagnostic I use to check whether a projector is truly multimodal and whether its outputs land in the correct embedding manifold. It’s not a replacement for the full graph, but it helps detect incomplete mmproj files and geometry drift. import torch def check_mmproj(mod): """ Checks whether the mmproj contains: - Vision tower - Audio tower (mel + STFT + FastConformer) - Video temporal embedder """ report = {} # Vision report["vision"] = "v.patch_embd.weight" in mod # Audio (FastConformer + mel + STFT) audio_keys = [ "a.mel_filterbank.weight", "a.stft.window", "a.encoder.layers.0.self_attn.q_proj.weight" ] report["audio"] = all(k in mod for k in audio_keys) # Video (temporal patch embedding) report["video"] = "v.patch_embd.weight.1" in mod return report def geometry_compatibility(backbone_dim, projector_out): """ Checks whether projector outputs lie in the correct embedding manifold. If mean cosine collapses or variance spikes, the projector is drifting. """ if projector_out.shape[-1] != backbone_dim: return "dimension mismatch" sims = torch.nn.functional.cosine_similarity( projector_out.unsqueeze(1), projector_out.unsqueeze(0), dim=-1 ) return { "mean_cosine": sims.mean().item(), "std_cosine": sims.std().item() } # Example: # mod = torch.load("Nemotron-Omni.mmproj", map_location="cpu") # print(check_mmproj(mod)) # print(geometry_compatibility(2688, torch.randn(32, 2688)))

NVIDIA’s Nemotron-3-Nano-Omni-30B is a genuinely multimodal model: image, audio, and video in. But until now the GGUF ecosystem only carried its vision half. The existing GGUF repos ship vision-only projectors, and the video path did not exist in llama.cpp at all. I built the missing pieces and released the whole package: **Models:** engram-ae/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF · Hugging Face Nine quantizations from Q3_K_M to BF16, plus a unified projector file that carries both encoder towers (C-RADIO vision and Parakeet audio). One llama-server instance takes every modality on /v1/chat/completions, including the built-in web UI: drag in a photo, a wav, or an mp4. **Code:** GitHub - VincentKaufmann/llama.cpp-omni: llama.cpp fork: Parakeet/FastConformer audio graph for Nemotron-3-Omni (mtmd) · GitHub A llama.cpp fork with the Parakeet/FastConformer audio graph and the C-RADIO video graph (temporal patches plus EVS token pruning), including one-pass video WITH its soundtrack: the model describes what it sees and quotes what it hears from a single request. Prebuilt Linux arm64 CUDA 13 binaries in the releases (DGX Spark, Jetson Thor). **Verification, since claims are cheap:** * The video graph matches NVIDIA’s PyTorch reference to 0.0019 percent relative L2, with exact EVS selection parity * The audio graph transcribes the test narration word for word * Every quantization was tested on image, audio, and video with soundtrack before upload. The graded matrix ships in the repo as TEST-MATRIX.md, so you can see exactly what was checked rather than taking my word for it Everything was built and tested on a single DGX Spark (GB10). Quick start is in the model card: one server flag pair, then open the browser. Roadmap: the next release rebases onto current llama.cpp master and adopts the upstream audio projector layout, so image, text and audio will run on stock llama.cpp with these same files; the fork remains for video and one-pass A/V. An upstream PR for the video path is planned. Base model and encoders are NVIDIA’s, under the NVIDIA Open Model Agreement. I wrote the GGUF conversion, the llama.cpp graphs, and the projector unification. Happy to answer questions or fix what breaks.

The Sticky Thistles played an outdoor set on a spring Sunday in the hills of Berkeley, California. Dynamic vocals, ripping string solos and harmonies. The beauty of old bluegrass favorites made new ag

Successful tests in Europe will see the pair deploy Omni-Path and eventual Ethernet-based platforms

Merrell and Big Brothers Big Sisters of America have partnered together to help clear a path to a child's biggest possible future with a uniquely designed Moab Speed 2 that will raise funds and awareness for this youth-mentoring non-profit organization. Merrell will donate $10 from every purchase to support Big Brothers Big Sisters.

Intel spinout CEO makes the case for Omni-Path as it toasts Top500 supercomputer spot

Intel-born Omni-Path networking revived by Cornelis Networks as 400Gbps alternative to Nvidia InfiniBand for DoE supercomputers.

Omni-Path lights up Lawrence Livermore system at 400 Gbps

Omni-Path lights up Lawrence Livermore system at 400 Gbps

Omni-Path lights up Lawrence Livermore system at 400 Gbps