Red Hat AI | 3 | Red Hat Documentation

Red Hat AI | 3 | Red Hat Documentation

KServe, the leading standardized AI inference platform on Kubernetes, has been accepted as an incubating project by the Cloud Native Computing Foundation (CNCF).

KServe, the leading standardized AI inference platform on Kubernetes, has been accepted as an incubating project by the Cloud Native Computing Foundation (CNCF).

Open Source Distributed AI Inference (llm-d/vLLM) Meetup Boston/Cambridge Hosted by Google Cloud, Red Hat AI, and the llm-d Community Date: Thursday, May 28th…

Hi everyone, I am currently running **RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic** using vLLM through Docker. My current vLLM command is: vllm serve --model RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic \ --trust-remote-code \ --attention-backend TRITON_ATTN \ --max-num-seqs 4 \ --max-model-len 8192 \ --gpu-memory-utilization 0.78 \ --generation-config vllm \ --hf-overrides '{"diffusion_sampler":"entropy_bound","diffusion_entropy_bound":0.1}' \ --diffusion-config '{"canvas_length":256}' \ --host 0.0.0.0 \ --port 8085 I am using the same model/server in two different applications: 1. A **RAG-based chatbot** 2. A **question-generation application** Interestingly, the model works reasonably well in the RAG chatbot. I don’t see noticeable grammar problems or word-merging issues there. However, in the **marine-domain question-generation application** , I sometimes get outputs with: * words being merged together * missing spaces * grammatical errors * malformed or unnatural questions For example, I can get output similar to: > `upcomingbunkering operation` instead of: > `upcoming bunkering operation` I also see cases where the generated question structure or grammar is not correct. The important part is that **the same model and vLLM server are being used** , but the problem is much more noticeable in the question-generation workflow. I understand that DiffusionGemma is experimental and that its output quality may not be comparable to standard autoregressive Gemma models. However, I am trying to understand whether the behavior I’m seeing is actually a model limitation or whether there is something different in my question-generation pipeline that I am missing. Could the difference be caused by things such as: * Prompt structure * Input/context length * Generation/sampling parameters * Chat template or thinking configuration * `canvas_length=256` * `entropy_bound` / `diffusion_entropy_bound` * FP8 quantization * Output formatting/constraints * Post-processing/tokenization * Differences in how the two applications call the vLLM API Has anyone experienced similar **word-merging or grammar issues with DiffusionGemma specifically during question generation or structured output generation**? Any suggestions on what I should compare or test between the RAG and question-generation pipelines would be appreciated. I would particularly like to know whether this is expected behavior from the diffusion generation approach or whether there is a configuration issue I should investigate.

Move beyond RAG to agentic AI with Model Context Protocol. See how Red Hat OpenShift AI secures and scales MCP servers for reliable enterprise automation

Discover the financial benefits and return on investment (ROI) experienced by customers using Red Hat AI. Learn how organizations turned infrastructure challenges into measurable financial gains with a 3-year ROI of 233% and a 13-month payback period.

Red Hat Brings Distributed AI Inference to Production AI Workloads with Red Hat AI 3

Learn how Red Hat OpenShift AI's ISO 42001 certification reinforces Red Hat's leadership in responsible AI, providing enhanced customer data protection, industry standard alignment, and platform maturity.

Discover the financial benefits and return on investment (ROI) experienced by customers using Red Hat AI. Learn how organizations turned infrastructure challenges into measurable financial gains with ...

YouTube video by Red Hat

Red Hat Brings Distributed AI Inference to Production AI Workloads with Red Hat AI 3