NERVOSYS — AI & Robotics

Every serving loop, training runtime, and agent dispatch system sitting on top of a GPU wants the same thing at the bottom of the stack: a Rust interface over the vendor driver that's fast, correct, and unsurprising. On CUDA, that's usually cudarc — the de facto choice, and a good one. But its per-call thread-context verification and per-buffer event-fence Drop add real overhead in a hot dispatch loop: roughly 350 nanoseconds per allocation, 30 nanoseconds per sync. At hundreds of allocations a second — KV-cache slots, per-token scratch — that adds up.

Today we're announcing IronAccelerator: a low-level, hardware-agnostic Rust interface over NVIDIA, AMD, Apple, Qualcomm, Intel, Google, and AWS accelerators, plus the cross-vendor compute APIs (Vulkan, Direct3D 12, OpenGL, WebGPU). It's a drop-in replacement for cudarc's driver and NVRTC surface — same API shape, one-line use swap — and it's faster on every host-side hot path we measured.

IronAccelerator is dual-licensed under AGPL-3.0-or-later, with a commercial option available from Nervosys.

The 30-second port

// before
use cudarc::driver::{CudaDevice, CudaSlice, LaunchAsync};
use cudarc::nvrtc::compile_ptx;

// after — one import line
use ironaccelerator_cuda::cudarc_compat::{CudaDevice, CudaSlice, LaunchAsync, compile_ptx};

let dev    = CudaDevice::new(0)?;
let stream = dev.default_stream();
let xs     = stream.htod_copy(vec![1.0f32, 2.0, 3.0])?;
let out    = stream.dtoh_sync_copy(&xs)?;

Everything past the import line is identical to what you'd write against cudarc 0.19. The cudarc_compat module doc ships a side-by-side coverage map, so an agent porting a cudarc codebase can answer "where did foo go" without reading source.

Why it's faster

Four rules, held consistently across the whole crate:

Cache the driver function table inline. Every Device, Stream, Event, and Module stores a &'static DriverFns reference resolved once at construction. A wrapped call reaches the function table via a struct-field load — no atomic, no thread-context rebind. cudarc calls cuCtxGetCurrent on every driver call to verify the bound context; IronAccelerator binds once at Device::open and trusts it to persist.
Cold-path every error. The success path compiles to load/test/branch with no Error-enum materialisation; the failure branch is #[cold] #[inline(never)].
Kill eager allocations on the hot path. An overflow-check helper was heap-allocating a String on every successful call. Replacing it with a cold helper alone took a 64 KB allocation from 491 ns to 340 ns.
Inline the public surface. Every accessor, every Drop, every wrapped driver call is #[inline], so the whole alloc path collapses to about 20 instructions before the FFI call.

MemPool: the headline win

For a dispatch loop that allocates and frees hundreds or thousands of buffers a second, the cuMemAllocAsync round trip itself becomes the bottleneck. MemPool is an opt-in, per-stream freelist that recycles DeviceBuf allocations into power-of-two byte buckets:

use ironaccelerator_cuda::pool::MemPool;

let pool = MemPool::new(stream.clone());
for _ in 0..n {
    let buf = pool.alloc::<f32>(1024)?;   // pop bucket, no FFI
    // ... use buf in kernels exactly like any DeviceBuf ...
    drop(buf);                              // push bucket, no FFI
}

The warm path is a thread-local front cache — an UnsafeCell access plus a fixed-array index, no FFI, no mutex — at roughly 10 nanoseconds per alloc/free cycle. Cross-thread overflow spills to a shared, mutex-guarded cache; over-cap requests fall through to the driver. Against cudarc's warm-path allocation, that's ~70× faster, and PooledBuf<'p, T> derefs to DeviceBuf so every existing method, including the whole cudarc_compat surface, works unchanged.

The numbers, measured back-to-back

On an RTX 3090 Ti (CUDA 13.2), against cudarc 0.19.6, host→device wins at every size tested, CI-confirmed: at the small end (≤1 KiB, ~1.29×) the win is pure per-call overhead — one cached-pointer copy versus cudarc's clone-then-synchronize path — dipping to ~1.08–1.10× in the 256 KiB–1 MiB PCIe-bound band where the transfer itself dominates, then climbing to ~1.12× through 16 MiB and ~1.28× at 64 MiB as a pinned-staging path takes over (the driver can't DMA out of pageable memory on a non-null stream, so it stages internally — a 16 MiB transfer dropped from ~36 ms pageable to ~1.6 ms on a quiet device).

Device→host is at parity across the whole sweep, and we say so rather than round it up. Both libraries land in a tight band around the driver floor — measurement noise, not a code difference. Seven different approaches were implemented and measured before concluding this; each is documented at its call site so it isn't re-attempted.

On the control plane — where a thin wrapper's cost is the cost, because the underlying driver call is cheap — the wins are consistent: stream synchronize at ~1.5×, event create/record/sync/destroy at ~1.5×, async alloc/free at ~1.9–2.0× across sizes. Kernel launch is a correction, not a win: we originally reported ~1.45× here, measured on a machine that wasn't actually idle. Re-measured with nothing else scheduled on the box, both libraries land at ~4.6–4.7 µs — parity. A raw cuLaunchKernel call is cheap enough, and both wrappers route to it directly enough, that there's no real wrapper-level gap there; the earlier number was contention noise, and we're not leaving it uncorrected.

The honest backend matrix

CUDA is the only backend that's production-ready today. Every other backend — ROCm, Metal, Vulkan, QNN, OpenGL, Direct3D 12, WebGPU, TPU via PJRT, Level Zero, AWS Neuron — compiles, registers, and enumerates devices where the vendor SDK is present, but none of them has had the optimization pass that got CUDA's MemPool to ~70× over cudarc. Detailed per-backend gap analysis lives in the repo's own STATUS.md, not buried in a changelog. If you're shopping for a drop-in cudarc replacement, the CUDA backend is the whole point of the project today; everything else is scaffold with a real path to parity, not vaporware.

Every backend loads its vendor library via libloading at first use — the workspace builds on any host, with or without the SDK installed. A missing runtime surfaces as a typed NotAvailable error, never a link failure.

Designed for agents

Errors name the operation. Error::Driver { op: "cuMemAllocAsync", code } — no anonymous CUDA error payloads. An agent can grep the source for the op string and land directly on the call site instead of doing stack-trace archaeology.
No domain code. IronAccelerator stops at the driver layer — no kernels, planners, FP8 recipes, or workload autotuners. The surface is small enough that an agent can hold the whole API in context; that scope boundary is deliberate, and everything above it lives in IronWorks.
One way to do each thing. No five flavors of allocation, no two stream types, no three launch APIs.

Production use: IronWorks

IronWorks, our Rust LLM inference engine, completed a full migration off cudarc onto IronAccelerator on 2026-05-15 — roughly 300 call sites, dropping cudarc from Cargo.lock entirely, with zero kernel regressions. Token generation on Llama-3.2-1B Q4_K_M held at ~522 tokens per second, within 1.3% of the cudarc baseline: IronWorks is kernel-bound, so the wrapper-level wins amortize to noise there, but the migration validated that every CUDA-driver call site IronWorks needs has a native IronAccelerator equivalent.

Try it

[dependencies]
ironaccelerator-cuda = "1.1"

# Full workspace
git clone https://github.com/nervosys/IronAccelerator
cargo build --release

# Just the CUDA backend
cargo build --release -p ironaccelerator-cuda

# Reproduce the benchmarks yourself
cargo bench -p ironaccelerator-cuda --bench vs_cudarc
cargo run --release -p ironaccelerator-cuda --example saxpy_cudarc_style

IronAccelerator: github.com/nervosys/IronAccelerator — AGPL-3.0-or-later + commercial

For commercial licensing (embedded, proprietary, or closed-source SaaS use cases), see LICENSING.md in the repo.

What's next

The scope cut here was deliberate: workload descriptors, quantization, the heuristic planner, and the accelerator ontology all moved to IronWorks, leaving Backend as a discovery-only trait. What's left on the CUDA side is a Rust-side small-buffer free list to skip cuMemFreeAsync round trips at very high churn. What's left everywhere else is real hardware: HIP FFI for ROCm, Metal/MPS bindings via objc2, QNN SDK FFI, and tighter Level Zero capability probing — each gated on getting the target hardware into CI so the same honesty bar applies to every backend, not just CUDA.

Per aspera ad astra.