Apple and NVIDIA have teamed up in research to improve generative AI performance, but specifically generating text with LLMs.

It’s a different implementation, but it seems like TensorRTLLM is easier to use with TensorRT… huggingface.co ### Accelerated inference on NVIDIA GPUs We’re on a journey to advance and democratize artificial intelligence through open source and open science. github.com ### GitHub - huggingface/optimum-nvidia Contribute to huggingface/optimum-nvidia development by creating an account on GitHub. huggingface.co ### TensorRT-LLM backend We’re on a journey to advance and democratize artificial intelligence through open source and open science. Or try newer torch-tensorrt? github.com/pytorch/TensorRT #### docker/README.md `main` # Building a Torch-TensorRT container * Use `Dockerfile` to build a container which provides the exact development environment that our main branch is usually tested against. * The `Dockerfile` currently uses <a href="https://github.com/bazelbuild/bazelisk">Bazelisk</a> to select the Bazel version, and uses the exact library versions of Torch and CUDA listed in <a href="https://github.com/pytorch/TensorRT#dependencies">dependencies</a>. * The desired versions of TensorRT must be specified as build-args, with major and minor versions as in: `--build-arg TENSORRT_VERSION=a.b` * [**Optional**] The desired base image be changed by explicitly setting a base image, as in `--build-arg BASE_IMG=nvidia/cuda:11.8.0-devel-ubuntu22.04`, though this is optional. * [**Optional**] Additionally, the desired Python version can be changed by explicitly setting a version, as in `--build-arg PYTHON_VERSION=3.11`, though this is optional as well. * This `Dockerfile` installs `cxx11-abi` versions of Pytorch and builds Torch-TRT using `cxx11-abi` libtorch as well. As of torch 2.7, torch requires `cxx11-abi` for all CUDA 11.8, 12.4, 12.6, and later versions. Note: By default the container uses the `cxx11-abi` version of Torch + Torch-TRT. If you are using a workflow that requires a build of PyTorch on the PRE CXX11 ABI, please add the Docker build argument: `--build-arg USE_PRE_CXX11_ABI=1` ### Dependencies * Install nvidia-docker by following https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker ### Instructions - The example below uses TensorRT 10.9.0.34 This file has been truncated. show original