Post cover image
Table 1. The five models as served on the DGX Spark. Memory is the size of the weight files on Hugging Face; the KV cache for a 64K context and runtime buffers come on top, and everything shares the Spark's 128 GB with the CPU and, during investigations, the incident databases.

November 5, 2026

Selecting a model for local agentic AI inference in SecOps on NVIDIA DGX Spark

This is my next round of NVIDIA DGX Spark testing, following NVIDIA DGX Spark on My Desk: A Builder’s Review of the Local Inference Machine…

By Mirek Fokt

5 min read

This is my next round of NVIDIA DGX Spark testing, following NVIDIA DGX Spark on My Desk: A Builder's Review of the Local Inference Machine I'd Been Waiting For. Thanks to TD SYNNEX, the distributor that lent us the machine.

That first review ran GPT-OSS 120B next to a full private voice RAG stack in the Spark's 128 GB, at about 40 tokens/s. It argued that larger models make agentic tool calling reliable enough to use. This time we moved toward a production use case and put that claim to a measured test.

We wanted to find open-source models suitable for an AI SOC, with an NVIDIA DGX Spark as the inference hardware. Every model had to run with a 64K-token context window, to leave room for longer and more demanding agentic tasks. So we designed a test to answer two questions: how fast does each model serve on the Spark, and how well does it handle tool calling and agentic work when solving SOC investigations from a public benchmark?

We tested five models. Two came out ahead: Nemotron 3.5 Super with LiLiCorr speculative decoding answered 55 of 60 investigations correctly in 1h52m, and Qwen3.6 answered 51 in 1h23m.

Raw speed was a poor guide to the real work. Nemotron 3.5 Lightning was the fastest model on the synthetic benchmark but answered only 38 correctly. GLM-5.3-Flash answered 54 but needed more than six hours.

Speculative decoding was no free win either. It made Super 2.4× faster on the investigations, but it made Mistral 82% slower on the inference test.

Setup

Each model ran alone on one DGX Spark, a GB10 machine with 128 GB of memory shared by CPU and GPU. Each served an OpenAI-compatible API, one request at a time, with a 64K-token context for the investigations. Table 1 lists the five models and how we served them.

We chose mixture-of-experts (MoE) models because of the Spark's memory bandwidth. Its 128 GB is large but comparatively slow, at 273 GB/s, and every generated token means reading the active weights from memory. An MoE model reads only a fraction of its weights per token: Lightning, for example, activates 3B of its 30B parameters.

None

Lightning and Super were set up from NVIDIA's DGX Spark serving recipes for Lightning and Super, with 64K context and one sequence.

Speculative decoding pairs the model with a small drafter. The drafter proposes several tokens, the main model checks them in one pass, and it keeps the ones it accepts. For Super it took two extra flags and no new image.

The test had two parts, one for each question:

  • Inference speed: vllm bench serve with 50 random chat prompts of 8,192 input and 256 output tokens, one at a time, after three warmups. GLM was served by llama.cpp at 32K context, with vLLM's benchmark client.
  • Agentic SOC investigations: 60 tasks from ExCyTIn-Bench, run through Microsoft's ACESEvals harness. The model investigates by querying incident databases through tool calls, up to 25 per task, then answers. Claude Opus 5 grades the answers.

Inference speed

Lightning and Qwen3.6 were the fastest at 8K input and 256 output tokens (Figure 1): about 4 s per response and about 60 tokens/s. GLM took 40 s per response, 29 s of it before the first token. All 400 requests across the eight configurations succeeded.

None

Speculative decoding sped up generation, not prompt reading. LiLiCorr cut Super's time per token by 41% and GLM's MTP head cut it by 33%, but neither changed time to first token. For GLM, where prompt reading dominates, responses got only 12% faster.

Draft acceptance decided whether it paid off. LiLiCorr produced 3.52 tokens per verification step and DSpark 2.35.

Speculative decoding made Mistral slower. Its EAGLE drafter had only 9.5% of drafts accepted, so checking the drafts cost more than they saved, and responses took 82% longer.

That run used locally patched vLLM and returned 8% fewer tokens than requested, so treat it as indicative. Mistral's investigations ran without EAGLE.

Agentic SOC investigations

Super + LiLiCorr and Qwen3.6 gave the most correct answers for the time (Figure 2). Qwen finished 26% sooner than speculative Super, with four fewer correct answers. GLM came within one answer of Super but took 3.3× as long.

None

Lightning was the second-fastest run but had the fewest correct answers. It wrote 79% more output tokens than speculative Super, and made twice as many model requests as baseline Super (1,367 vs 686). Mistral Small 4 answered two more than Lightning in 60% more time.

LiLiCorr cut Super's investigation time 2.39× with the same 55 correct answers. Qwen3.6 and GLM used a revised tool-use prompt, and the other models used the original. Part of their result may come from the prompt rather than the model.

What we learned

  1. Benchmark the agent loop, not just the server. Lightning generated 2.5× as many tokens per second as speculative Super. Yet it finished the 60 investigations only 4 minutes sooner, with 17 fewer correct answers.
  2. Check tool calling before scoring a model. Qwen3.6 first scored 0 of 3 on a smoke test: it printed SQL after Action: instead of calling tools. Replacing the prompt's Thought: / Action: format with native tool calls brought it to 3 of 3. Its chat template also needed a fix to accept several system messages.
  3. Measure speculative decoding; don't assume it. It made Super 2.39× faster on investigations and Mistral 82% slower on inference. Draft acceptance is the number to watch.
  4. Prompt length sets the floor on response time. Super's time to first token tripled from 0.96 s at 2K input to 2.95 s at 8K. Investigation requests carried 6.7K–10.9K input tokens at the median and 22K–36K at p95, so prompt reading speed counts at every step.
  5. 128 GB fits big models, with care. GLM's 2-bit weights alone fill 85% of the Spark's memory, and Super's NVFP4 weights 59%. The investigation databases share the same memory, so headroom has to be checked per workload.
  6. Read speculative-decoding metrics carefully. Streamed chunks carry several accepted tokens, so inter-token latency misleads: Lightning showed 24.95 ms against a mean time per output token of 10.71 ms. vLLM also reported a peak concurrency of two in runs capped at one, so we ignored that field.

Verdict and next steps

Super + LiLiCorr and Qwen3.6 move on to testing with our own tools and cases. Lightning stays as a speed reference, and GLM as a quality reference at far higher latency. On the first review's claim, size helped but was not everything: the 120B-class Super had the most correct answers, and the 35B Qwen3.6 came within four answers with a revised prompt.

These are first results, not a ranking:

  • Each configuration ran once, so there are no repeat runs or error bars.
  • The 60 tasks come from two of the benchmark's eight incidents.
  • Every run fixed a malformed MySQL password example in the upstream prompt, and Qwen3.6 and GLM also used the revised tool-use prompt. Grading used Claude Opus 5, so scores are not comparable to published ExCyTIn-Bench results.
  • Engines, quantization and provider settings differ, so these results compare deployments, not models alone. Investigation time includes tool calls and grading.

Next we will:

  • Rerun every model on the same prompt, with repeat runs across more incidents.
  • Test the models on more demanding benchmarks, both public ones and our own, built from our own tools and analyst-reviewed cases.
  • Measure how much a better prompt and agent harness can lift each model. A prompt change alone took Qwen3.6 from 0 to 3 of 3 on its smoke test.
  • Test longer prompts (16K and 32K) and concurrency above one.

Sources