We talk about CoDesign through the lens of the river and sometime you have to jump in to swim. But in many cases understanding the conditions first and foremost is critical.

Nvidia says AI memory pressure is rising as HBM supply tightens. Its full-stack codesign plan uses NVFP4, Dynamo and KV cache tiering to cut memory needs.

This PhD project aims to explore how digital technology can empower people with cerebral palsy (CP) and their families to manage their health, be active partners in healthcare decision-making and navi...

NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory.

Larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise — a trend expected to continue through the near term. NVIDIA works closely with memory partners to rapidly adopt the latest memory technologies. NVIDIA was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack. But the approach isn’t to simply pack more memory onto the GPU. It’s full-stack codesign. NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory. That matters because an AI factory runs on a memory and storage hierarchy. GPU high-bandwidth memory (HBM) is the fastest, most expensive and most contested tier — with model weights and KV cache competing for limited capacity. Below it sit CPU memory, local SSDs and shared storage, each trading speed for more capacity at a lower cost per bit. How much useful work a byte does depends on how well the whole hierarchy is used: shrinking what must be stored, placing it on the right tier and bringing it back to the GPU before it’s needed. Byte for byte, NVIDIA makes the most efficient use of memory at every layer — and the results show it. From NVIDIA Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3%, while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX. ## **Agentic Inference Is Putting More Pressure on HBM** Frontier models now have trillions of parameters — the values that hold what an AI model learned in training. KV cache, which holds the context the model needs to understand each active request without reprocessing the same information, is also expanding. Agentic trajectories carry prior context across repeated model and tool calls, in some cases processing up to 3 million tokens, so each request’s working memory grows over time. Retrieval-augmented generation (RAG) adds relevant enterprise information to prompts, creating even more context to process and retain. During inference, HBM primarily holds both model parameters and KV cache. The result: greater pressure on HBM. ## **Why AI Factories Need a Memory Hierarchy** Adding more HBM capacity per GPU is the obvious response to that pressure, but capacity delivers only what bandwidth can move. Think of HBM as a warehouse and memory bandwidth as its loading docks. A larger warehouse holds more inventory, but the docks set how much can move in and out at a time. Once the docks are at capacity, moving more inventory takes longer and slows response times. If delays exceed the required service level, the added space can’t be used effectively, even as it adds cost and draws power. That’s also why no operation keeps all its inventory in the closest, fastest space. What’s in motion stays near the line. What’s waiting moves to a larger, cheaper space nearby and returns before it’s needed. An AI factory faces the same limit. More HBM can hold more KV cache and support more concurrent users. Every added user shares the same compute and memory bandwidth. But when each user expects fast responses, eventually memory bandwidth is unable to supply data as fast as compute can use it: a limit engineers call the memory wall. Past that point, extra HBM capacity can’t be put to work, yet it still adds costs and draws power. The data still needs a place to live, but not all of it needs the fastest, most expensive tier. This is why an AI factory manages data across a memory and storage hierarchy. ## **Maximizing the Value of Every Byte With Full-Stack Codesign** NVIDIA’s full-stack codesign makes every byte of HBM count by relieving pressure from model weights, KV cache and RAG data. It does this with techniques that each target a different source of inefficiency: **Shrink model weights and distribute them across the NVL72 domain.** * The NVFP4 format reduces the space required for weights compared to FP8, the lowest precision possible on the NVIDIA Hopper platform. Recent models, including FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark from Meta, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL have been released with NVFP4 checkpoints, demonstrating growing model-builder adoption of 4-bit precision for faster, more memory-efficient inference. * Within an NVL72 system, model weights can be distributed across up to 72 GPUs, so each GPU holds a smaller share of the model — leaving more HBM for KV cache and concurrent users. **Shrink KV-cache and offload inactive context to lower-cost storage tiers.** * Applying NVFP4 to the KV cache reduces its footprint compared with FP8, so more context can stay resident in HBM. * NVIDIA Vera CPU helps in two ways. Faster tool execution shortens the time KV cache sits inactive during tool calls, and its large LPDDR memory, connected to NVIDIA Rubin GPUs over NVIDIA NVLink-C2C interconnect, gives paused context a nearby place to wait and return quickly. * When context isn’t actively in use, NVIDIA Dynamo inference software moves it to NVIDIA CMX context memory storage — a tier built for capacity at far lower cost per bit than HBM — and returns it before reuse. Unlike local SSDs, CMX is shared across servers, and unlike enterprise storage, it skips the overhead of long-term data protection that short-lived context doesn’t need. NVIDIA CMX context memory storage partners include Cloudian, DDN, Dell Technologies, Everpure (previously Pure Storage), HPE, IBM, NetApp, VAST and WEKA. * Dynamo’s KV Cache Reuse ties the tiers together, managing KV cache as one system-wide resource across GPU memory, CPU memory and storage, so context computed once can be reused across the system. **Compact RAG embeddings and keeping the index outside HBM.** * RAG turns enterprise documents into embeddings — numerical representations that let a system search by meaning rather than keyword — organized into a searchable index. NVIDIA Nemotron Retriever models produce smaller embeddings with higher retrieval accuracy, while the NVIDIA cuVS library enables that index to be built and served across CPU and storage. The knowledge base never occupies HBM, and only the passages the model needs enter context. The result is less HBM used for model weights, less HBM tied up by idle context and more of it available to serve concurrent users. ## **More Useful Output per Dollar With Each Generation** Across successive NVIDIA platform generations, tokens per second per dollar has risen far faster than HBM capacity. On the SemiAnalysis AgentX DeepSeek V4 Pro workload, NVIDIA GB300 NVL72 delivers up to 10x higher throughput per dollar than NVIDIA H200 platform at 100 tokens per second per user, with HBM capacity per GPU roughly doubling. At 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, with HBM capacity per GPU up about 3%. For AI factory operators, the metric that matters is how many useful tokens the factory can deliver within its power budget while meeting user response-time needs. More tokens per second per watt means more work and revenue from the same power, and a lower cost for every token. NVIDIA’s full-stack codesign lowers token costs by increasing the useful work every byte of memory delivers. _Learn more in the_ _video explainer_ _and_ _technical deep dive_ _on how NVIDIA full-stack codesign reduces inference memory pressure._

Larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise — a trend expected to continue through the near term. NVIDIA works closely with memory partners to rapidly adopt the latest memory technologies. NVIDIA was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack. […]

Background: AI technologies have considerable potential to support mental health practitioners in identifying depression symptoms. However, existing tools are often developed with limited practitioner involvement and evaluated primarily on technical performance, leaving questions of clinical relevance, ethical design, and practitioner trust insufficiently addressed. Objective: This study aimed to examine how iterative co-design with mental health practitioners can inform the development of an AI-based model for identifying depression symptoms, focusing on how practitioner input shaped its clinical relevance and ethical foundations across 2 design cycles. Method: We conducted a 2-stage qualitative co-design study. In stage 1, we developed DepressionSymp 1.0, a fine-tuned GPT-3.5 model trained on the CounselChat dataset, identifying 5 symptom categories based on the () criteria and the Linguistic Inquiry and Word Count (LIWC-22) psycholinguistic framework, reviewed and discussed by 6 mental health practitioners during online co-design workshops. In stage 2, this feedback informed DepressionSymp 2.0, expanded to 10 symptom categories and reviewed by 8 further practitioners in a second round of workshops. Results: Practitioners valued the structured symptom outputs for reducing cognitive workload and supporting intervention planning. In stage 1, indirect indicators such as social withdrawal, financial difficulties, and major life changes were often classified as lifestyle factors rather than symptoms; this informed the expansion to 10 categories in stage 2, described as clearer and more useful for preassessment. Risk categorization was identified as the most ethically sensitive aspect across both stages, alongside broader concerns including the lack of nonverbal information in text-based assessments, cultural and linguistic differences in symptom expression, and the potential for automation bias without sufficient human oversight. Conclusions: Iterative co-design with mental health practitioners is valuable for developing AI-based tools that are clinically relevant and ethically grounded, demonstrating how qualitative practitioner feedback can inform iterative model refinement and ethical design. Findings inform design implications around large language model (LLM) features, practitioner-in-the-loop oversight, and ethical considerations covering clinical standards, informed consent, and data governance. Future work should prioritize larger-scale independent validation, formal safety evaluation, and culturally diverse dataset development.

In less than 3 hours (at 11:00 UK time) we will be meeting to codesign what we want #CriticAI community to be. If you want to share your thoughts about how #AI is used in #Academia join us here: 📆 Today, 11:00 (UK Time) 📔 Notes: hackmd.io/psmbmkxaSJWg-PoqYn... 📍 Jitsi: meet.jit.si/moderated/0d68aa...

In less than 3 hours (at 11:00 UK time) we will be meeting to codesign what we want #CriticAI community to be. If you want to share your thoughts about how #AI is used in #Academia join us here: 📆 Today, 11:00 (UK Time) 📔 Notes: hackmd.io/psmbmkxaSJWg-PoqYn... 📍 Jitsi: meet.jit.si/moderated/0d68aa...

In less than 3 hours (at 11:00 UK time) we will be meeting to codesign what we want #CriticAI community to be. If you want to share your thoughts about how #AI is used in #Academia join us here: 📆 Today, 11:00 (UK Time) 📔 Notes: hackmd.io/psmbmkxaSJWg-PoqYn... 📍 Jitsi: meet.jit.si/moderated/0d68aa...