Deployment Shapes
Three configurations, one engine. Single-stream interactive runs
a single sequence through the resident kernel — lowest possible latency, no per-token host
involvement, deterministic given a seed. Bucketed batch server
groups concurrent requests by length via dynamic-programming-optimal bucketing and processes each
bucket as one pass through the kernel, amortizing setup across every request in the bucket — the
right shape for workloads with fairly uniform generation lengths: translation, summarization,
structured generation, classification. Continuous-batching server
holds a fixed set of sequence slots inside the same resident kernel and keeps them occupied: a
sequence that hits its stop condition, its length cap, or an eviction retires at the next admission
boundary, and a waiting request is admitted into the freed slot without the kernel ever exiting.
Admission and retirement in the continuous-batching shape happen on a fixed device-side cadence
rather than at every single decode step, which keeps the coordination cost off the critical path
without ever leaving the kernel to pay a relaunch cost. A late-arriving request joins a batch
already in flight rather than waiting for the current one to drain. High-variance chat workloads,
where generation lengths differ by an order of magnitude across concurrent requests, are served
without falling back to host-side rescheduling — the trade is a small, bounded admission latency
and fixed-size per-slot KV allocation in exchange for keeping the entire batching decision
on-device.
Per-token streaming is available on every request in every configuration. Partial output is emitted
to the client as tokens retire from the egress ring rather than held until a sequence completes.
Why: Cathedral Architecture
The founding insight is simple to state: in a transformer forward pass, only
batch size and sequence length are runtime-mutable. Everything
else — layer count, head dimensions, hidden sizes, quantization scheme, tensor parallelism degree —
is knowable ahead of time. Most inference engines treat this as runtime configuration anyway,
paying for flexibility they don't need with dispatch overhead they can't get back.
Nihilus takes the opposite position. A binary built for up to 20 model configurations only runs
those configurations — the trade-off is a rebuild per model collection, in exchange for moving nearly
everything a conventional engine resolves at runtime into build time instead. That single decision
compounds through the rest of the architecture: work that produces no computation is designed out
rather than executed and discarded, the memory hierarchy is used deliberately rather than left to
the hardware's default caching behavior, and the server runs start to finish without handing
control back to the host between tokens.
Architecture Highlights
- One persistent GPU kernel for the lifetime of the server — prefill, decode, admission and retirement all run without returning to the host
- Continuous batching is resolved on-device: finished sequences retire and new ones take their slots at the next admission boundary, with no kernel relaunch
- The forward pass is decomposed into a fixed set of fused stages, each with a deliberate memory-traffic boundary
- Per-layer structure is resolved ahead of time rather than iterated at runtime
- A compact, build-time-derived metadata structure drives execution with near-zero runtime lookup cost
- On-chip memory is used as a deliberate scratchpad between stages, not left to opportunistic caching
- Cross-GPU synchronization is unified across single- and multi-GPU builds from the same source
- Cross-GPU communication happens from device code directly — no host-coordinated collective operations
- Single-GPU and multi-GPU deployments share one codepath, differing only in what's compiled in
- Tokens leave the kernel through a device-side egress ring, drained and streamed by a host thread that never stalls the kernel
- Each fused stage gets independently tuned resource allocation rather than sharing a worst-case budget
- Data-transfer pipelining is tuned per stage to its own resource footprint
- Low-level memory transfer paths are hand-tuned for the exact semantics each boundary requires
- Quantization is handled as a first-class, swappable precision — 8-bit today, with a clear path to others
Discipline
The same discipline that governs Jsonifier
applies here. A short list of unsafe or ambiguous C++ constructs is banned outright, each ban tied
to an architectural property the engine depends on rather than kept as a style preference. Where
low-level reinterpretation is genuinely required, it goes through a single, defined-behavior
wrapper rather than the unsafe casts most C++ codebases rely on.
The codebase compiles warning-clean under the strictest configuration each compiler supports,
across seven first-class toolchain targets spanning Windows and Linux, CPU and GPU, with
sanitizer coverage on every applicable path. GPU code compiles
through a single toolchain exclusively, with no boundary between host and device compilation to
introduce mismatches.
- Single-digit-second CPU build, low-double-digit-second GPU build
- Compact binaries relative to the model classes they serve
- Byte-level fuzz testing on input-handling paths, zero-crash / zero-undefined-behavior standard
- No host-device ABI boundary anywhere GPU code is involved
How Nihilus Compares
Honest positioning matters more than a highlight reel. Here's where Nihilus sits relative to the
engines a licensee would otherwise consider:
TensorRT-LLM
The closest architectural competitor. TRT-LLM fuses aggressively but still launches many kernels per forward pass with host-coordinated cross-GPU collectives, and its in-flight batching is orchestrated from the host between launches; Nihilus keeps one kernel resident and rebalances the batch inside it, with cross-GPU communication issued from device code. TRT-LLM has far broader model support and NVIDIA's own ecosystem behind it.
vLLM
Continuous batching and PagedAttention are mature and production-proven, and vLLM's scheduler is more configurable than ours today. The architectural difference is where the scheduling happens: vLLM rebalances from the host between kernel launches, Nihilus rebalances inside a kernel that never exits. PagedAttention trades an indirection on every attention read for KV blocks that float with actual sequence length, which buys concurrency density at fixed VRAM. Nihilus takes the other side: contiguous per-slot KV, no block table, no gather, with slot count fixed at build time. Which one wins depends on whether your bottleneck is per-token cost or concurrent sessions per card.
SGLang
RadixAttention's prefix-sharing is a real advantage for workloads with heavy prompt overlap — shared system prompts, few-shot examples, RAG with common context. Nihilus has no prefix caching in the current binary; this is a gap, not a wash.
llama.cpp
Different use case entirely. llama.cpp runs on nearly any hardware for single-user or small-scale deployment; Nihilus targets NVIDIA Hopper-and-newer production deployment at scale. The two barely compete for the same licensee.
TGI
Competes more on integration than raw inference performance — Hugging Face ecosystem compatibility, swappable backends, production serving features. A licensee optimizing for "fits our existing HF stack" leans TGI; a licensee optimizing for "lowest per-token cost on NVIDIA hardware" leans Nihilus.
Nihilus is best-positioned for production-scale Llama-family inference on Hopper-class or newer
hardware where per-token cost is the dominant operational concern. Outside that profile — heavy
prefix sharing, multi-architecture serving, non-NVIDIA hardware — one of the alternatives above is
likely the better fit, and we'd rather say so than pretend otherwise.
Status
Nihilus v1.0 is under active development. Continuous batching is implemented and running against
the persistent kernel; prefix caching remains on the roadmap. Out of scope for 1.0: speculative
decoding, multimodal inputs, LoRA adapters, and multi-architecture serving. The Cathedral
Architecture manifesto and further technical detail are available separately. Benchmark results
and licensing details will be published once the methodology and verification harness are ready
to stand on their own — consistent with how
Jsonifier's
benchmark results are presented on this site.
