voxflow uses Qwen3-TTS for local Chinese voice cloning and voice design, and it has always run on PyTorch + MPS. This time I moved it to Apple MLX, and every number comes from A/B tests on the same M3 Pro.
The Numbers First
Same model, same sentence, same reference audio — only the inference framework changed:
| Metric | PyTorch + MPS | Apple MLX 8-bit | Change |
|---|---|---|---|
| Model size (Base + VoiceDesign) | 8.4 GB | 5.8 GB | −31% |
| Inference time | 9.32 s | 5.29 s | 1.76× faster |
| Output duration | 4.64 s | 4.56 s | Identical |
| Audio quality (whisper transcription round-trip) | Not a single character off | Not a single character off | No loss |
| Peak memory | — | 6.97 GB (measured) | Documented |
After changing the code I ran the end-to-end test again: load 7.90s, inference 5.93s, output 4.40s — 2.35× faster than baseline.
Why MLX Is the “Right” Framework on a Mac
Apple Silicon solders the CPU, GPU, and memory onto a single chip, sharing one pool of unified memory. A traditional PC has one chunk of memory for the CPU and another chunk of VRAM for the discrete GPU, with data shuttling across PCIe in between.
That difference decides whether a local model runs smoothly:
| Apple M series | x86 + discrete GPU | |
|---|---|---|
| What runs the model | Unified memory (one pool for CPU/GPU) | VRAM |
| Does data get moved? | No, the CPU writes and the GPU reads directly | Yes, across PCIe every time |
| Not enough VRAM | The concept doesn’t exist here | It just crashes; your only option is a smaller quantized version |
But the real root cause was hiding in a single environment variable.
Root Cause: MPS’s Silent Degradation
The first line of voxflow’s old code was:
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
It means operators MPS doesn’t support silently fall back to the CPU, shipping the tensors back and forth along the way. All the benefit of “unified memory” is wasted at that step, and you can’t see it from the outside — clean logs, no warnings, just slow.
MLX has no fallback: either everything runs on the Metal GPU, or it errors. Silent degradation doesn’t exist.
That’s the biggest win from the migration: not just being 1.76× faster, but that “a GPU operator secretly running on the CPU” can no longer happen.
Three Pitfalls That Nearly Derailed Me
Pitfall 1: An empty ref_text = garbled output
The MLX base model’s clone mode requires ref_text (the transcription of the reference audio).
Leave it empty and you get no error — just truncated output with garbled content:
ref_text |
Output | ASR round-trip |
|---|---|---|
"" |
2.08s (truncated) | 「这是一字语音色争要了根根」❌ |
| Real text | 4.56s | 「这是一次语音合成测试…」✅ Not a single character off |
Fix: transcribe the sample audio with whisper and put it into personas.json. The cost: one extra required field per character — but what it buys is the cloning quality itself.
Pitfall 2: “Emotion instructions” were a capability I assumed, not one I was actually using
The PyTorch version of the base model accepts both x_vector_only_mode=True and instruct_ids.
Reading the source, I treated them as “two independent channels, both active”, and almost used that to conclude the migration was lossy.
Then I went through the configuration of all two characters in the project and found:
jxx_hostgoes through cloning and never relied on emotion instructions at alldemo_narratorgoes through the design path, and its instruction is a timbre description, not a dynamic emotion
The question “is emotion control a hard requirement” is answered by looking at the data — no character in the project uses it. A capability matrix derived from reading code loses to going through the real configuration once.
Pitfall 3: Dependency conflict — transformers 4 vs 5
mlx-audio wants transformers>=5.14, the project pins ==4.57.3, and the two don’t fit in the same venv.
My first instinct was “keep PyTorch as a fallback” — later I realized that fallback was a net liability:
- Every change required confirming the old path still worked
- A/B tests had to run twice
- And MLX had already been verified end to end
Delete it. MLX became the only backend, and core/engine.py went from 122 lines down to 61.
Rolling back went from “git checkout three files” to “git revert one commit” — the PyTorch weights stay where they are, ready whenever.
Migration Cost and Verification
Under 50 lines changed, concentrated in engine loading plus 3 call sites:
# MLX 的 API 跟 PyTorch 几乎同签名
results = list(model.generate(
text=text,
ref_audio=seed_path, # 文件路径直接传
ref_text=ref_text, # 必填
lang_code="chinese",
temperature=0.7, top_k=50, top_p=0.9,
))
audio = results[0].audio # mx.array
Verification method: transcribe both versions’ output back to text with whisper — not a single character off is the pass bar. “Sounds about the same” doesn’t count — an ASR round-trip as objective comparison is stricter than an A/B test by ear.
What This Actually Means for an 18GB Mac
迁移前：TTS 8.4G + ASR 1.5G + 视觉 3.0G = 12.9G ← 18G 机器爆边缘
迁移后：TTS 5.8G + ASR 1.5G + 视觉 3.0G = 10.3G ← 留出余量
The 2.6 GB freed up by TTS is exactly the key that lets “voice + image recognition” stay resident at the same time. Lazy loading and idle unloading are no longer optimizations; they’re a hard constraint that couldn’t be met before.
When You Shouldn’t Migrate
- Not Apple Silicon — MLX is Mac-only
- Model > unified memory — MLX’s advantage comes from zero-copy; if it doesn’t fit in memory, all of that goes to zero
- Running in the cloud — CUDA is more mature there, and MLX makes no sense
- You need operators MPS has already tuned — run an A/B on your real workload before migrating; don’t trust my numbers
Reflecting on This Round
The most valuable thing wasn’t that comparison table; it was the two judgments that got overturned along the way:
- “AI image generation writes the wrong numbers, so AI can’t draw technical comparison graphics” — gpt-image-2 slapped that down on the first try, getting the Chinese text and every number right in one shot. Using an old impression as a conclusion without verifying it was my biggest mistake this round.
- “The capability matrix comes from reading the source” — no matter how carefully you read, it loses to going through the project’s real configuration once. Both “nearly lossy migration” misjudgments died from not looking at the data first.
Don’t guess, measure. The time it takes to run one A/B test is shorter than arguing for an afternoon.
