The Setup

The Setup

We run a small but powerful distributed inference cluster: five Apple Mac Studio machines (each with 96 GB unified memory), connected over Thunderbolt 5 and running open-weight LLMs via MLX. The cluster looks like this:

• ai001–ai005 on a private 192.168.101.x subnet
• Thunderbolt 5 mesh for inter-node communication
• jaccl as the RDMA backend — Apple’s Thunderbolt-based RDMA transport for macOS 26
• mlx.launch for distributed inference across nodes
• launchd (macOS’s init system) running each model as a system service with KeepAlive

Our largest model, GLM-4.6–5bit, runs distributed across four nodes. QwQ-32B-8bit runs across two. A custom shell script, mlx-preseed.sh, handles pre-seeding model weights to peer nodes before launching the server.

Everything was great — until it wasn’t.

The Symptom

The Symptom

Every day, without fail, we’d get alerts:

LLM model glm-4_6-5b is down (firing, severity: critical)
LLM model qwq-32b-8bit is down (firing, severity: critical)

The services wouldn’t come back on their own. Manual intervention — sometimes a full node reboot — was required. We’d fix it, go to sleep, and wake up to the same alerts.

The logs showed a cryptic error during restart:

RuntimeError: [jaccl] Couldn't allocate protection domain
RuntimeError: [jaccl] Couldn't allocate protection domain
RuntimeError: [jaccl] Couldn't allocate protection domain

Every retry failed with the same error. The models were stuck.

At some point the jokes started. I suggested to mount a robot arm to the server cabinet, hook it up to the Prometheus webhook, and have it physically unplug and replug the Thunderbolt cables every time an alert fired. We laughed. Then we realized we were seriously considering it. That’s when we decided to actually figure out what was going on.

The Diagnosis: A Cascade in Three Acts

The Diagnosis: A Cascade in Three Acts

Act 1: The mlx.launch Teardown Bug

When mlx.launch starts a distributed job, it SSHes into each peer node and launches a rank process there. When it exits (either cleanly or due to a crash), it's supposed to kill those remote ranks.

Here’s what actually happens in the teardown:

# from mlx/_distributed_utils/launch.py
subprocess.CalledProcessError: Command 'ssh ai005 'pid=$(cat /var/folders/.../tmp.None);
  if ps -p $pid >/dev/null; then kill $pid; echo 1;
  else echo 0; fi; rm /var/folders/.../tmp.None'' returned non-zero exit status 1.

The tmpfile path is tmp.None. The substitution failed — the pidfile variable was never set. The remote rank processes are never killed.

Act 2: Orphan Accumulation

launchd's KeepAlive means the service restarts immediately after a failure. Each restart:

1. mlx-preseed.sh runs
2. Calls mlx.launch with the RDMA hostfile
3. mlx.launch fails (or the job fails for any reason)
4. Remote rank processes on peer nodes are left behind — orphaned
5. launchd restarts the service in seconds
6. Repeat

After enough restarts, the peer nodes accumulate hundreds of orphaned rank processes. On ai002, we saw over 100 of them:

ai    71234  0.0  0.0  ...  Us+  serve.py --model mlx-community/GLM-4.6-5bit ...
ai    71891  0.0  0.0  ...  Us+  serve.py --model mlx-community/GLM-4.6-5bit ...
ai    72103  0.0  0.0  ...  Us+  serve.py --model mlx-community/GLM-4.6-5bit ...
# ... 97 more

The state is Us+: uninterruptible sleep, session leader, foreground. These processes are stuck in a kernel-level wait inside the jaccl RDMA stack. The processes that accumulated in Us+ state cannot be killed — not even with kill -9. The original sweep did successfully kill the small subset with ppid=1, because those processes, by definition, had an sshd parent that had already exited cleanly — meaning they were never in Us+ state to begin with. The vast majority of orphans, the Us+ ones with a stuck sshd parent, were never even attempted.

Act 3: Protection Domain Exhaustion

Every mx.distributed.init() call allocates a jaccl protection domain — an RDMA resource representing an isolated communication context. These domains are finite. Each orphaned process holds one and never releases it — and as we’d later learn, that’s compounding a leak that already exists on every teardown cycle.

Once the pool is exhausted, every new attempt fails immediately at init, before any actual work is done:

RuntimeError: [jaccl] Couldn't allocate protection domain

The service is now permanently broken until someone reboots the node and flushes all processes from memory. A 17-hour-old orphaned rank process from the previous day was holding a domain the current job needed.

Why the Existing Cleanup Missed This

Why the Existing Cleanup Missed This

mlx-preseed.sh already had a pre-flight sweep to kill orphaned processes before starting. But it had a critical blind spot:

for pid in $(pgrep -f "mlx-share-rank.py.*--tmpdir ${tmpdir}"); do
    [ "$(ps -o ppid= -p "$pid" | tr -d '[:space:]')" = "1" ] || continue
    kill "$pid"
done

The ppid=1 check was meant to identify orphans: when a process's parent dies, it gets reparented to init (PID 1). The logic was sound — except for one macOS-specific behavior.

On peer nodes, rank processes are launched via SSH by mlx.launch. Their parent is sshd. When mlx.launch crashes and closes the SSH connection, those remote processes get stuck in Us+ state — and here's the key: the sshd subprocess itself also gets stuck waiting for its child to exit. Neither process dies. Neither gets reparented to init. The orphaned rank's ppid is a stuck sshd process, not 1.

The sweep looked for ppid=1, found little to nothing, and moved on. Every time.

It took a few more down-alert cycles before the pattern became clear: the sweep wasn’t failing outright, it was just quietly losing ground. It caught the occasional legitimately-reparented process, but the Us+ orphans — the ones actually driving us toward exhaustion — kept slipping through every single restart.

The Fix

The Fix

1. Fix the SWEEP

The fundamental insight: at sweep time, mlx.launch hasn't been called yet for this restart. No legitimate mlx-share-rank.py process can exist for this model on any node — the previous run's supervisor is already gone. The ppid=1 check was both wrong (missing sshd-parented orphans) and unnecessary (the --tmpdir filter already scopes the kill to this model only).

# Before: misses peer orphans
for pid in $(pgrep -f "mlx-share-rank.py.*--tmpdir ${tmpdir}"); do
    [ "$(ps -o ppid= -p "$pid" | tr -d '[:space:]')" = "1" ] || continue
    kill "$pid"
done

# After: catches killable orphans; those already in Us+ survive but stop accumulating
for pid in $(pgrep -f "mlx-share-rank.py.*--tmpdir ${tmpdir}" 2>/dev/null); do
    echo "[preseed] cleanup: killing orphaned mlx-share-rank.py pid=$pid"
    kill -9 "$pid" 2>/dev/null
done

A process becomes orphaned immediately when mlx.launch crashes, but it doesn't enter Us+ instantly — it gets there gradually as it tries to access RDMA resources that are no longer available. The sweep runs at the very start of each restart, catching orphans in that window while they're still killable. Processes already deep in Us+ will survive the sweep, but the sweep prevents new ones from accumulating on each restart cycle.

The same logic applies to serve.py orphans — but here we keep one guard: if something is actually listening on the port, we never touch it regardless of process state. An active server must not be disrupted.

if [ -n "$port" ] && ! /usr/sbin/lsof -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then
    for pid in $(pgrep -f "serve\.py .*--port $port" 2>/dev/null); do
        echo "[preseed] cleanup: killing orphaned serve.py pid=$pid"
        kill -9 "$pid" 2>/dev/null
    done
fi

This sweep runs on every node in the hostfile — local and all peers — before anything else happens.

2. Fix the Race Condition

We sometimes run mlx-preseed.sh manually from the terminal when diagnosing a down model. Without protection, a manual run and launchd's concurrent restart would both execute the sweep simultaneously — each killing the other's active rank processes.

A simple PID-file lock prevents this:

LOCKFILE="/tmp/mlx-preseed-${SHORT}.pid"
if [ -f "$LOCKFILE" ] && kill -0 "$(cat "$LOCKFILE" 2>/dev/null)" 2>/dev/null; then
    echo "[preseed] already running. Use 'launchctl kickstart -k system/com.valensas.vale.${SHORT}'"
    exit 0
fi
echo "$$" > "$LOCKFILE"

# ... preseed work happens here ...

rm -f "$LOCKFILE"
exec serve.py ...

The lock is released just before exec serve.py, not after — because exec replaces the shell process entirely, and any trap EXIT wouldn't fire. When serve.py exits and launchd starts a fresh preseed, the lock file is already gone.

3. Clean Up Stale Temporary Directories

mlx_lm.share stages model weights into a temporary directory during transfer. If a rank is killed hard, Python never gets to clean it up. Over weeks of failures, peer nodes had accumulated hundreds of tmp* directories — 817 GB on one node, 868 GB on another.

# Runs after the sweep, on local node and all peers
find "$CACHE_DIR" -maxdepth 1 -name 'tmp*' -exec rm -rf {} + 2>/dev/null || true
for peer in $PEERS; do
    ssh -o BatchMode=yes -o ConnectTimeout=5 "$peer" \
        "find '${CACHE_DIR}' -maxdepth 1 -name 'tmp*' -exec rm -rf {} +" || true
done

Order matters: sweep first (kill processes), then clean directories. Not the other way around.

4. Prevent GPU Timeouts from Triggering the Cascade

A separate but related problem: QwQ-32B was occasionally crashing with Metal GPU timeouts:

libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed: Caused GPU Timeout Error
(00000002:kIOGPUCommandBufferCallbackErrorTimeout).

This would trigger the whole cascade — crash → teardown bug → orphan accumulation → protection domain exhaustion.

The cause: without --prefill-step-size, MLX processes the entire input prompt in a single Metal command buffer. For long prompts, this exceeds Apple's GPU timeout threshold.

One parameter in the inventory:

extra_params:
  prefill-step-size: 2048

This chunks prefill into 2048-token batches, each fitting comfortably within Metal’s limits.

The Result

The Result

Before our changes, the failure chain looked like this:

jaccl failure
    → mlx.launch teardown bug → orphans on peers
    → KeepAlive restart → more orphans
    → protection domain pool exhausted
    → permanent failure until reboot

After:

jaccl failure
    → mlx.launch teardown bug → orphans on peers
    → KeepAlive restart → SWEEP kills killable orphans
    → clean restart
    → service recovers

We still see the occasional down alert — a crash is still a crash. But the difference is what happens next: instead of staying down until someone notices and reboots a node, the service now self-heals. The sweep catches the orphans before they can pile up, so each restart actually has a chance to succeed instead of failing the same way forever.

This turned out to be a known, unresolved issue: ml-explore/mlx-lm #955 documents the same root cause — JACCL protection domains aren’t fully released on teardown, and “there is no other way to reclaim PDs” besides a reboot. Our fix doesn’t patch that kernel limitation; it removes the trigger (orphaned processes from crash-restart loops) that was pushing us into exhaustion in hours instead of, apparently, much longer. One week in: zero reboots, self-heals on the crash.

Key Takeaways

Key Takeaways

1. ppid=1 is not a reliable orphan detector on macOS. When a parent process gets stuck in uninterruptible sleep waiting for a child, the child keeps the parent's PID — not init's. The scope filter (matching by --tmpdir or port) is the real guard, not the ppid check.

2. Us+ processes on macOS cannot be killed — only prevented. SIGKILL does nothing to a process in uninterruptible sleep. The sweep's value is catching orphans before they enter Us+ state, not killing them after. On each KeepAlive restart, freshly orphaned processes are still reachable — the sweep catches those. Once a process is truly stuck, only a reboot flushes it. Prevention, not cure.

3. KeepAlive can turn a transient bug into a permanent failure. A single orphan is harmless. Fifty orphans accumulated over rapid restarts exhaust a finite resource pool and make every subsequent attempt fail immediately, no matter how healthy the actual environment is.

4. Teardown bugs in distributed launchers compound quickly. The mlx.launch teardown issue would be a minor annoyance in a manually-managed cluster. In a KeepAlive-managed service, it becomes a time bomb.

5. Order matters in cleanup scripts. Kill processes before deleting their working directories. It’s obvious in retrospect, but easy to get wrong under pressure.

We’re running mlx 0.32.0, macOS 26, and jaccl on Thunderbolt 5 hardware. Some of this is very early-days infrastructure — the RDMA stack in particular is still maturing. If you’re running similar setups, hope this saves you a few reboots.