Meta, the project's maintainer, shrugs

Researchers with Israeli firm Oligo published information about three critical issues with TorchServe, a part of the PyTorch project overseen by Amazon and Meta. The code helps companies build AI models into their businesses.

Ray Serve DLCでTorchServeデプロイを効率化

TorchServe is no longer maintained, leaving teams to own the entire GPU inference stack. The AWS Ray Serve Deep Learning Container is a supported, pre-tested container with the framework, GPU drivers, and serving layer already assembled. This post walks through deploying a vision-language model on Amazon EKS using the Ray Serve DLC on a single GPU node.

TorchServe is no longer actively maintained, so AWS ships a pre-built Ray Serve container for model inference.

AWS launches Ray Serve Deep Learning Containers to simplify model inference deployment, offering a maintained alternative to the deprecated TorchServe with pre-optimized GPU stack and dependencies.

TorchServe is no longer maintained, leaving teams to own the entire GPU inference stack. The AWS Ray Serve Deep Learning Container is a supported, pre-tested container with the framework, GPU drivers, and serving layer already assembled. This post walks through deploying a vision-language model on Amazon EKS using the Ray Serve DLC on a single GPU node.

TorchServe is no longer maintained, leaving teams to own the entire GPU inference stack. The AWS Ray Serve Deep Learning Container is a supported, pre-tested container with the framework, GPU drivers, and serving layer already assembled. This post walks through deploying a vision-language model on Amazon EKS using the Ray Serve DLC on a single GPU node.

A clutch of vulnerabilities in an open source tool used by major corporations to scale up machine learning models could lead to remote takeover, says a

The vulnerabilities exist in the widely used TorchServe framework, used by Amazon, Google, Walmart, and many other heavy hitters.

Critical vulnerabilities named "ShellTorch" have been found in the popular TorchServe AI model-serving tool maintained by Meta and Amazon, affecting versions 0.3.0 through 0.8.1 and posing a significant risk to tens of thousands of internet-exposed servers, including those of major organizations. The vulnerabilities include an unauthenticated management interface API misconfiguration, a remote server-side request forgery (SSRF) flaw, and a Java deserialization problem, potentially allowing unauthorized access and malicious model uploads. Users are urged to upgrade to TorchServe 0.8.2 and implement specific configurations to mitigate these risks.

Multiple vulnerabilities in TorchServe, used for serving PyTorch models, could lead to remote code execution.

ShellTorch attack chains critical TorchServe vulnerabilities and could completely compromise the AI infrastructure of major companies.

It seems to be an unsolved issue. It looks like it could be used as it is… github.com/pytorch/serve #### examples/README.md `master` # Examples showcasing TorchServe Features and Integrations ## Security Changes TorchServe now enforces token authorization and model API control by default. This change will impact the current examples so please check the following documentation for more information: Token Authorization, Model API control ## TorchServe Internals * Creating mar file for an eager mode model * Creating mar file for torchscript mode model * Serving custom model with custom service handler * Serving model using Docker Container * Creating a Workflow * Custom Metrics * Dynamic Batch Processing * Dynamic Batched Async Requests ## TorchServe Integrations ### Kubernetes <img src="images/k8s.png" width="30" title="K8s" style="float:right padding:10px" /> This file has been truncated. show original github.com/pytorch/serve #### Update Huggingface_Transformers examples for safetensors opened 07:01PM - 12 Nov 23 UTC nathanweeks triaged security ### 📚 The doc issue In examples/Huggingface_Transformers/README.md, the `torch-…model-archiver` commands specify `--serialized-file Transformer_model/pytorch_model.bin` option-arguments. However, the `Download_Transformer_models.py` script in the same directory currently results in a downloaded model in the safetensors format: ``` $ python Download_Transformer_models.py ... Save model and tokenizer/ Torchscript model based on the setting from setup_config bert-base-uncased in directory ./Transformer_model $ ls -1 Transformer_model/ config.json model.safetensors special_tokens_map.json tokenizer_config.json tokenizer.json vocab.txt ``` ### Suggest a potential alternative/fix Perhaps update the examples to accommodate both formats.

현재 llama2 모델을 A100 GPU를 사용해서 서빙해보려고 합니다. mig로 구성된 인스턴스 4개를 사용하려고 하는데(single node, gpu 4장, 1장 당 mig 인스턴스 1개 <- 7g80GB), mig 인스턴스의 gpu 여러장을 활용해서 분산 추론을 서빙하고 싶은데.. 여기저기 확인해보니 mig 컨셉상 이렇게 활용하는게 아닌 것 같더라구요;; 그렇다 해도 혹시 방법이 없을까 찾아보고 있는데.. torchrun으로 4개로 sharding된 모델파일을 각각 gpu의 올리고 추론하는것은 해봤는데요 (nccl이 불가능해서 gloo 사용), torchserve에서는 또 이게 어떻게 할지를 모르겠어서.. 조언이 필요합니다.

TorchServe의 입력 유효성 검사가 없는 이슈를 이용하여 원격으로 디스크에 파일을 다운로드 받을 수 있는 보안 이슈가 발견되었습니다. 이는 TorchServe 0.8.1 이하 모든 버전에서 발견된 이슈로, 0.8.2 버전으로의 업데이트를 권고드립니다. (관련 PR #2534) TorchServe에서 공지한 보안 이슈는 아래와 같습니다. 주요 내용은 아래와 같습니다. (feat. DeepL) 원격 서버 측 요청 위조(SSRF) / Remote Server-Side Request Forgery (SSRF) 이슈: TorchServe 기본 구성에는 적절한 입력 유효성 검사가 없어 제3자가 원격 HTTP 다운로드 요청을 호출하고 디스크에 파일을 쓸 수 있습니다. 이 문제를 악용하여 시스템과 민감한 데이터의 무결성을 손상시킬 수 있습니다. 이 문제는 0.1.0~0.8.1 버전에 존재합니다. Issue : TorchServe default configuration ...

Multiple vulnerabilities in TorchServe, used for serving PyTorch models, could lead to remote code execution.