diff --git a/.github/workflows/build-self-hosted.yml b/.github/workflows/build-self-hosted.yml index dd31b1d15..9b4f5d294 100644 --- a/.github/workflows/build-self-hosted.yml +++ b/.github/workflows/build-self-hosted.yml @@ -115,7 +115,7 @@ jobs: vulkaninfo --summary GG_BUILD_VULKAN=1 bash ./ci/run.sh ~/results/whisper.cpp ~/mnt/whisper.cpp - amd-npu-windows: + npu-amd-windows: runs-on: [self-hosted, Windows, X64, stx, rai300-400] timeout-minutes: 60 continue-on-error: true # advisory while the runner pool is new; revisit later @@ -208,7 +208,7 @@ jobs: name: vitisai-smoke-log-windows path: vitisai.log - amd-npu-linux: + npu-amd-linux: runs-on: [self-hosted, Linux, X64, stx, rai300-400] timeout-minutes: 60 continue-on-error: true # advisory while the runner pool is new; revisit later diff --git a/README.md b/README.md index fd23e1c55..18f37d563 100644 --- a/README.md +++ b/README.md @@ -314,47 +314,80 @@ This can result in significant speedup in encoder performance. Here are the inst For more information about the OpenVINO implementation please refer to PR [#1037](https://github.com/ggml-org/whisper.cpp/pull/1037). -## AMD Ryzen™ AI support for NPU +## AMD Ryzen™ AI NPU support -On AMD's Ryzen™ AI 300 Series with dedicated NPUs for acceleration, you can now run Whisper models with the ability to fully offload the encoder to NPU. This brings significant speedup compared to CPU-only. -> **Note:** -> **Ryzen™ AI NPU acceleration is currently supported on Windows only.** Linux support is planned for upcoming releases. -> For the latest updates on Ryzen AI, check out [the official documentation](https://ryzenai.docs.amd.com/en/latest/). +On AMD Ryzen™ AI 300 and 400 Series processors with a dedicated NPU, whisper.cpp can fully offload the Whisper encoder to the NPU via VitisAI, delivering significant speedup over CPU-only inference. -### Setup environment (Windows only) +### Prerequisites - - Obtain the XRT package and the FlexmlRT package from AMD. Both are distributed as tarballs or wheels. - - Copy the downloaded archives to a local path, extract them, and run the setup script from each extracted package in your shell (for example `source /path/to/xrt/setup.sh` and `source /path/to/flexmlrt/setup.sh`). Run these in every new shell you use to build or run `whisper.cpp`. +Install the XRT runtime and FlexML runtime for your platform: -- Fetch the matching ggml model and prebuilt VitisAI encoder cache: +- **XRT**: provides the NPU kernel driver and `xrt-smi` diagnostic tool +- **FlexML runtime** (`flexmlrt`): VitisAI inference engine used by whisper.cpp — download from the [FlexML runtime releases](https://github.com/lemonade-sdk/whisper.cpp-rocm/releases/tag/deps) - ```bash - sh ./models/download-ggml-model.sh base - sh ./models/download-vitisai-model.sh base - ``` +After installing, source the setup scripts in every shell you use to build or run whisper.cpp: - ```cmd - .\models\download-ggml-model.cmd base - .\models\download-vitisai-model.cmd base - ``` +```bash +# Linux +source /opt/xilinx/xrt/setup.sh +source /path/to/flexmlrt/setup.sh +``` - Use the same model name with both scripts. The VitisAI script queries the AMD collection on Hugging Face to list available caches, then downloads the selected `.rai` file as `ggml--encoder-vitisai.rai` alongside the matching `ggml-.bin` file. You can also browse the collection manually at https://huggingface.co/collections/amd/ryzen-ai-whisper-npu-optimized-onnx-models. +```cmd +:: Windows +cd /path/to/flexmlrt && call setup.bat +``` - Depending on the downloaded `.rai` cache, VitisAI may offload either the encoder only or the encoder plus cross-projection layers to the AMD NPU. `whisper.cpp` detects the cache contents at runtime and logs the selected offload mode during model initialization. +You can verify the NPU is visible with: -- Build `whisper.cpp` with VitisAI support: +```bash +xrt-smi examine +``` - ```bash - cmake -B build -DWHISPER_VITISAI=1 - cmake --build build -j --config Release - ``` -Your environment is now ready. +### Download models -### Build Whisper.cpp for Ryzen™ AI support +Download the ggml model and the matching prebuilt VitisAI encoder cache: - ```text - $ ./build/bin/whisper-cli -m models/ggml-base.bin -f samples/jfk.wav - ``` +```bash +# Linux / macOS +sh ./models/download-ggml-model.sh base +sh ./models/download-vitisai-model.sh base +``` + +```cmd +:: Windows +.\models\download-ggml-model.cmd base +.\models\download-vitisai-model.cmd base +``` + +Use the same model name with both scripts. To see all available VitisAI encoder caches: + +```bash +sh ./models/download-vitisai-model.sh --list +``` + +```cmd +.\models\download-vitisai-model.cmd --list +``` + +The VitisAI script queries the [AMD Ryzen AI Whisper NPU collection on Hugging Face](https://huggingface.co/collections/amd/ryzen-ai-whisper-npu-optimized-onnx-models) and downloads the `.rai` encoder cache as `models/ggml--encoder-vitisai.rai`. + +> Depending on the `.rai` cache, VitisAI may offload the encoder only, or the encoder plus cross-projection layers. whisper.cpp detects this at runtime and logs the selected offload mode during model initialization. + +### Build + +```bash +cmake -B build -DWHISPER_VITISAI=1 +cmake --build build -j --config Release +``` + +### Run + +```bash +./build/bin/whisper-cli -m models/ggml-base.bin -f samples/jfk.wav +``` + +For more information see the [Ryzen AI documentation](https://ryzenai.docs.amd.com/en/latest/). ## NVIDIA GPU support