好文集ChatGPT Plus¥119

The ultimate battle: Four DGX Sparks 128GB vs. a Mac Studio M5 Ultra 512GB: in depth testing

文章目录

EDIT: The bar has been raised. See the new article: https://x.com/krunkosaurus/status/2121641060262895653

TLDR: On GLM-5.3-Flash at maximum reasoning, a ring of four DGX Sparks beat a 512 GB Mac Studio M5 Ultra at every prompt length from 2K to 1M tokens. It cut the wait for the first token by 13–28%, generated 1.3–1.8× faster, and delivered 1.8× the total throughput with four users on 8K and 32K prompts. Yet the Mac has more memory bandwidth than all four Sparks combined, and most of its generation deficit looks like software that is still catching up.

When I bought a Mac Studio M5 Ultra this week, I said I was still 100% team DGX Spark. I own seventeen Sparks, so I'm hardly neutral. I decided to test the claim properly: the same model, the same prompts and the same sampling settings on both machines, from a 2K-token prompt to a million tokens.

文章配图

The two machines.

The Spark ring. Four ASUS Ascent GX10s, NVIDIA's DGX Spark design, each with 128 GB of unified memory, cabled into a closed ring with 200 Gb/s links. TensorFold splits GLM-5.3-Flash across all four boxes. The experts are quantized to 4-bit EXL3, the dense layers are cut to 4-bit at load, the KV cache is stored in FP8, and a small DFlash2 model drafts tokens ahead of the main model for speculative decoding.

The Mac. One Mac Studio M5 Ultra with 512 GB of unified memory, running oMLX built from source with Seth's GLM-5.3-Flash recipe. The weights are a mostly 4-bit oQ4e build, the KV cache is uncompressed, and the model's own multi-token prediction (MTP) head does the drafting.

At list price the two setups cost about the same. @seapy put them at $25,799 for the Mac and $24,000 for four GX10s. I found this pretty accurate.

Keeping it fair.

Both machines received identical prompts, byte for byte: eight public-domain novels for the prose tests and CPython's standard library for the code tests, cut at exact token counts. A unique tag at the front of each prompt kept either machine from reusing a cached prefix. Sampling used the model's own defaults (temperature 1.0, top-p 0.95) at maximum reasoning, with replies capped at 8,192 tokens, or 2,048 in the multi-user tests. The same script ran on each machine's own localhost, timed every streamed token and checked each server's request counters for outside traffic. All 168 test blocks passed on the first attempt.

The setups are not identical, and I didn't force them to be. Each runs the recipe its community recommends, and the ring's has had weeks more tuning, plus two unpublished patches for tool calls, keep-alives and fault handling. Before testing, the Mac matched the recipe's published prefill figures and its 62-token-per-second prose decode test. Drafting depends on the text, so the reasoning-heavy runs below are slower. The ring's GPUs are locked at 2,200 MHz, below their boost clock, which if anything handicaps the ring.

Reading the prompt.

Before a model can answer, it has to read the entire prompt. That step is called prefill, and past 256K tokens it is most of the wait. From 8K tokens up, the ring reads 2,400 to 2,900 tokens per second and the Mac 1,700 to 2,260. At a 128K prompt the first token arrives after 46 seconds on the ring and 60 on the Mac. At a full million tokens it takes 7.2 minutes against 10.

文章配图

Generating the answer.

Generation is where the gap is widest. At a 2K prompt the ring generates 85 tokens per second on code and 64 on prose. The Mac manages 52 and 47. Up to 256K the ring holds a lead of 1.3–1.6×, and past that the lead grows. At a million tokens the ring still generates 71 tokens per second on code, while the Mac has slowed to 39.

Code makes the cleaner comparison at the long end, because every code reply was still reasoning when it reached the cap, while most prose replies at 768K and 1M had moved on to writing the story. Across the full range on code, the ring slows by 16% and the Mac by 24%.

文章配图

Sharing the machine.

With four people sending 8K prompts at once, the ring delivered 97 tokens per second in total on prose and 109 on code. The Mac delivered 53 and 59. The first person in line did better on the Mac, seeing a token in 4.0–4.2 seconds against the ring's 4.6–4.7. The fourth waited 18 seconds for a first token on the ring and 30 on the Mac, and at 32K prompts that wait grew to 72 seconds against 119. At 128K both machines spent about half their time reading other people's prompts, and the prose totals fell to 28 tokens per second on the ring and 20 on the Mac.

文章配图

Maximum reasoning, maximum length.

At its maximum reasoning setting, GLM-5.3-Flash began its answer in only 12 of 72 replies. The other 60 spent all 8,192 tokens reasoning, about 33,000 characters each. For this kind of work the speed you feel is generation speed: 8,192 tokens take about two minutes on the ring and three on the Mac. Against an earlier low-reasoning run, whose shorter replies got to the code, generation on code prompts ran about 10% slower on both machines up to 256K, likely because reasoning text is harder for a draft model to predict than code.

Where the Mac loses time.

On paper the Mac should at least keep up in single-user generation. Generation speed is limited by memory bandwidth, and the M5 Ultra has 1.23 TB/s, about 12% more than the four Sparks combined (4 × 273 GB/s), with no network between the parts. Its weights are heavier, though: about 4.5 bits per expert weight once MLX's scales are counted, against EXL3's 4.0, plus hundreds of tensors kept at 5–8 bits.

To see where the time goes, I split each machine's speed into two numbers: how long one pass through the model takes, and how many tokens each pass yields once the drafts are checked.

  • On prose, both machines average about two tokens per pass. The entire gap is pass time: 29 ms on the ring against 41 on the Mac at a 2K prompt, and 34 against 58 at a million.

  • On code, the ring's DFlash2 drafts yield 1.05–1.4× more tokens per pass, and its passes are faster as well.

  • From 2K to 1M tokens, the Mac's pass time grows about 40% and the ring's 10–17%. The Mac's uncompressed cache moves about twice the ring's bytes per cached token, which explains at most half of that extra slowdown.

  • With four users generating together on 8K prompts, the Mac's combined speed rises 24–30% over a single user. The ring's rises 71–80%.

Most of that looks like software to me. oMLX can change its drafter, cache format and batching, and even allowing for heavier weights, the Mac's passes run slower than its bandwidth predicts. I didn't measure bytes per pass, so that last point is an inference. Prefill is different. It is limited by compute, and four Blackwell GPUs bring more of it than one M5 Ultra. From 32K up, the Mac still reads prompts at 72–79% of the ring's pace.

What could change.

I expect the software side to improve in months rather than years. Longer drafters like DFlash2 can be ported to MLX, oMLX's compressed KV cache can be taught GLM-5.3-Flash's cache layout, and its batching will mature. Single-user generation should tighten considerably. Prefill and many-user throughput are less likely to flip at the same price without new hardware, and TensorFold keeps improving too.

There's no quick fix in the meantime. After the main runs I tried the three obvious settings on the Mac. Drafting three tokens ahead instead of two made generation 5–12% slower. The recipe's GPU memory-limit setting changed nothing beyond run-to-run noise, because the model never came near macOS's default cap. And oMLX's compressed KV cache doesn't support GLM-5.3-Flash yet, so it keeps the native cache and logs a warning. Seth's defaults were the fastest of everything I tried.

Today the ring is the faster machine at everything I measured for a single user, and it serves a team better. The Mac holds an entire million-token prompt in one box with about 265 GB of memory to spare, and its slower generation looks fixable in software. I'm still team DGX Spark. I'm also watching the Mac closely.

Run it yourself.

  • Spark ring: TensorFold 0.6.0 with Mia's AI Lab GLM-5.3-Flash EXL3 recipe (github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold, tp4 branch) and the Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold weights. My ring also runs the recipe's PR #205 and two patches that aren't public yet. The DFlash2 drafter (incoai/GLM-5.3-Flash-DFlash2) is licensed CC BY-NC-ND 4.0, for non-commercial use only.

  • Mac: oMLX built from source with Seth's recipe (github.com/sethforprivacy/GLM-5.3-Flash-M5-Ultra-oMLX, "recommended" profile) and the dfp-official/GLM-5.3-Flash-oQ4e-mtp weights with the recipe's MTP graft from Vontra/GLM-5.3-Flash-MLX-4bit-MTP (without it, oMLX skips the MTP head).

  • The full report, with every chart, table and setup detail: https://demo.sogni.ai/four-sparks-one-mac/

How I measured.

Each test ran twice per prompt length and prompt type, and repeats agreed within 5% on time to first token and within 10% on generation speed. Times are measured on the client, from sending the request to the first streamed token, which at maximum reasoning is a reasoning token. Generation speed counts the tokens after the first over the time they took to arrive, and the multi-user totals run from the first token any user sees to the last. The main runs skipped the Mac recipe's memory-limit setting (iogpu.wired_limit_mb), which needs an admin password. Capacity wasn't the issue in the single-user runs, which peaked at about 245 GiB at 1M against macOS's 480 GB GPU limit, and a rerun with the setting applied matched within run-to-run noise. These are speed numbers only: I didn't score answer quality, test long-context recall or measure power at the wall. On quality, the ring's 4-bit dense layers and FP8 cache are the bigger risk.

Thanks to Seth for the oMLX recipe, Mia's AI Lab for the TensorFold GLM build, ashhart for TensorFold, turboderp for EXL3, incoai for the DFlash2 drafter, jundot for oMLX, dfp-official and Vontra for the MLX weights, and Z.ai for GLM-5.3-Flash.

全文完
查看原文

沿着这个主题继续读

从一篇文章,找到相关的教程、经验和阅读路径。

本地大模型部署本地大模型部署与硬件选型,从模型参数、显存到文字、图像和 Agent 案例。