Strata 0.2.7.1 on GB10 Benchmark and SPEC_COST Ablation

I am not a native English speaker; this article was translated by AI.

Previously, I ran Strata on the DGX Spark. The machine features an ARM64-based GB10 with CUDA compute capability sm_121. To get Strata running on ARM64, I submitted PR #3622. In the merge commit, the maintainer noted that without physical ARM hardware on hand, the memory ordering was derived logically (“Memory ordering on ARM is reasoned, not run”). Testing on actual silicon, I verified the multi-threaded speculative verification with dmb ishld memory barriers across --spec 2 and --spec 4 windows, confirming byte-identical output consistency.

The maintainer quickly followed up with v0.2.7.1, bringing the underlying engine to a newer release. This tag directly provided official Linux ARM64 prebuilt archives (strata-engine-linux-aarch64-cuda-cuda13.tar.gz), added docs/DGX_SPARK_LINUX.md, and enabled four optimizations by default for sm_121:

• STRATA_PF_FUSED: Operator fusion during prefill
• STRATA_PREFILL_NVFP4: NVFP4 prefill targeting Blackwell architectures
• STRATA_HC_UPMIX: Mixed-precision channel up-mixing
• STRATA_SPEC_COST: Dynamic adjustment of speculative verification windows based on runtime cost and acceptance rate

The upstream release notes mentioned roughly a 1.20x–1.24x prefill speedup on the Spark. With the package published, I pulled it onto the machine to run an A/B benchmark.

Benchmark Setup

#

The runtime configuration remained identical: 262K context, INT8 KV cache, MTP, spec4, mmap experts, with all 24,576 experts resident in unified memory and no prompt cache.

To avoid cold-start discrepancies skewing the results, the tests followed a cross-validation sequence:

Legacy → Intermediate → New engine (with sm_121 optimizations disabled) → New engine with defaults → restart and reverse order.

Short requests used two warmups followed by three formal requests, confirming token output consistency. Long prompts were evaluated at 4K (4,099 tokens actual) and 16K (16,387 tokens actual).

The actual results showed two trends:

First, prefill was noticeably faster. Latency on 4K prompts dropped by nearly 20%, matching the 1.20x claim. At 16K, toggling hardware optimizations inside the new engine made little difference, but it was still over two seconds faster than the older builds.

Second, decode showed the opposite trend: short decode under default settings on the new engine ran a few hundred milliseconds slower than with optimizations disabled.

Testing SPEC_COST with ABBA

#

The decode slowdown naturally draws attention to STRATA_SPEC_COST, as it directly manages speculative window adjustment. Looking at a single benchmark run, it is easy to assume that the extra overhead slowed down generation.

On real hardware, however, variance across process starts can be considerable. To determine whether this switch caused the regression, I kept the identical binary and memory layout, toggling only STRATA_SPEC_COST=0 across four independent processes in an ABBA sequence (Default → Off → Off → Default):

Log inspection confirmed that the parameter was effective: with cost off, verification windows stayed fixed; with cost on, windows fluctuated dynamically within a range, evaluating more draft candidates.

Looking at decode time, however:

• From Process 1 to Process 2, disabling cost was about 2.5% slower;
• From Process 3 to Process 4, re-enabling cost was also about 2.8% slower.

This test eliminated a false hypothesis, preventing an inaccurate upstream report claiming that disabling SPEC_COST fixes decode latency.

Current Status

#

Having ready-made prebuilt packages and documentation for the Spark is very convenient, and all hardware-specific flags can be toggled off via environment variables, making testing straightforward.

The machine continues running the new version. The short decode gap is relatively minor for now, and I will isolate the impact of STRATA_HC_UPMIX and STRATA_PF_FUSED when time permits.