Most recent updates from several AI/ML model inference communities that our team at Red Hat AI is contributing to.

KServe, the leading standardized AI inference platform on Kubernetes, has been accepted as an incubating project by the Cloud Native Computing Foundation (CNCF).

KServe, the leading standardized AI inference platform on Kubernetes, has been accepted as an incubating project by the Cloud Native Computing Foundation (CNCF).

Ever waited too long for a model to return predictions? We have all been there. Machine learning models, especially the large, complex ones, can be painfully slow to serve in real time. Users, on the other hand, expect instant feedback. That’s where latency becomes a real problem. Technically speaking, one of the biggest problems is […]

Ye (Charlotte) Qi overviews LLM serving infrastructure challenges: fitting & speed (Model Runners, KV cache, and distributed inference), production complexities (latency optimization and continuous evaluation), and effective scaling strategies (heterogeneous deployment and autoscaling). Learn key concepts for robust LLM deployment. By Ye Qi

What's Changed Make storage initializer image configurable by @yuzisun in #3145 chore: Add design doc template links to feature request template by @ckadner in #3155 Increase pytest workers for ko...

Ray Serve DLCでTorchServeデプロイを効率化

What's Changed Make storage initializer image configurable by @yuzisun in #3145 chore: Add design doc template links to feature request template by @ckadner in #3155 Increase pytest workers for ko...

Practical patterns for scaling machine learning from your laptop to a distributed cluster.</b> Distributing machine learning systems allow developers to handle extremely large datasets across multiple clusters, take advantage of automation tools, and benefit from hardware accelerations. This book reveals best practice techniques and insider tips for tackling the challenges of scaling machine learning systems. In Distributed Machine Learning Patterns</i> you will learn how to: Apply distributed systems patterns to build scalable and reliable machine learning projects</li> Build ML pipelines with data ingestion, distributed training, model serving, and more</li> Automate ML tasks with Kubernetes, TensorFlow, Kubeflow, and Argo Workflows</li> Make trade-offs between different patterns and approaches</li> Manage and monitor machine learning workloads at scale</li> </ul> Inside Distributed Machine Learning Patterns</i> you’ll learn to apply established distributed systems patterns to machine learning projects—plus explore cutting-edge new patterns created specifically for machine learning. Firmly rooted in the real world, this book demonstrates how to apply patterns using examples based in TensorFlow, Kubernetes, Kubeflow, and Argo Workflows. Hands-on projects and clear, practical DevOps techniques let you easily launch, manage, and monitor cloud-native distributed machine learning pipelines.

Most recent updates from several AI/ML model inference communities that our team at Red Hat AI is contributing to.