No PyTorch: Building a Native HIP Inference Engine for AMD RDNA3 That Outperforms llama.cpp

This article is based on a LinkedIn post by Shisa AI CTO＆Co-founder Leonard Lin and is republished with permission.

Another "weekend" project I thought I'd share for those with AMD RDNA3 GPUs lying around: https://lnkd.in/grXNfXbM

hipEngine is a brand new open-source (AGPLv3) Python/CPP/HIP inference engine specifically tuned for gfx1100 (7900 XTX, W7900) and gfx1151 (Strix Halo) GPUs and currently runs a single model (Qwen 3.5 MoE ParoQuant), extremely fast. It does not have a PyTorch dependency, instead directly driving native HIP libraries, with 100+ (and growing) custom kernels.

The upshot to this is that while still a very early implementation (initial kernel optimization loops started at the beginning of the month, and the hipEngine harness itself is less than a week old), it's already performance competitive with llama.cpp on decode and beats it across the board on prefill (>2X faster at long context in some cases!).

I've also released my ROCm-compatible ParoQuant fork and a 19GB Qwen 3.6 35B-A3B PARO packed safetensor model): https://lnkd.in/gB7f5CdB

Developers may also be interested in reading some of the docs, including my gfx1100 roofline analysis doc: https://lnkd.in/gSCaTEWZ

The plan now that it's up and running is to give Qwen 3.6 a workout and have it continue to optimize itself whenever the GPUs have downtime. 😂

LinkedIn Original Post：