The Symptom Is Gone, Which Doesn't Mean the Premise Was Right — A Postmortem on Two Silent Failures

What both of today’s incidents had in common was a “silent success,” not a “loud failure.”

DSH heartbeats were wedged on a 401 by curl --fail, and all three layers of gigi’s failures were swallowed by >/dev/null 2>&1 — both ran for dozens of days, with blank logs and exit code 0.

What actually fixed it was those three lines in the sentinel: letting “it has never once succeeded” shout for itself, instead of waiting for someone to flip through the tables.

What Was Done and the Evidence

The sentinel already had an agent_stale alert, and it was running the whole time (it even reported dispatch_hub_stale at 03:50 today).

But gigi had zero heartbeats for two days and it never fired once. The cause is in cmd/watch.go:

hb := toString(a["heartbeat_at"])
if hb == "" {
    continue // 从未上报过心跳的 agent（如本机 CLI）不算失联
}

“Never reported” was treated as not being lost. And the justification given in the comment (“e.g. the local CLI”) had long since been caught by the runtime=="cli" check above it — so this fallback had only one remaining case it could hit:

Registered, has responsibilities, is a server agent, but was never once connected.

That is precisely the case that most deserves an alert.

Flipped it to an alert, with the wording kept distinct:

agent <id> has never reported a heartbeat since registration — credentials or the beat script never got connected

Added a test, TestNeverBeatenAgentAlerts, to pin this down: a server agent with an empty heartbeat must alert; cli:* must not. go test ./... is all green.

Three Things This Path Taught Me

① Pin down where the symptom belongs first, then follow it

At one point this morning I reported “DSH lost for 4h” to you as a fact and even proposed ssh-ing in to check — it was running right on this machine, and a single launchctl command would have disproved it.

A judgment is second-hand until you ask, “who computed this conclusion?”

② Before adding a field, check why the existing field is empty

kind couldn’t hold GUI, so my first reaction was to add a probe column; meanwhile the runtime column was sitting in the same table and printed Runtime: hermes in the same output, and I looked at it three times without stopping to ask “is this value right?”

An emptiness papered over by a default value looks exactly the same as genuinely not having the concept.

③ A patch removes the symptom; the premise stays in the code

When DSH was reported lost, I added an exception branch — “if the heartbeat is stale but the port answers, count it as alive” — and that still accepted the heartbeat as the primary criterion.

It took one line from you, “the local one doesn’t need this thing at all,” to replace the criterion itself.

The symptom is gone, which doesn’t mean the premise was right.

What You Must Understand Clearly

DSH heartbeats were wedged on a 401 by curl --fail, and all three layers of gigi’s failures were swallowed by >/dev/null 2>&1 — both ran for dozens of days, with blank logs and exit code 0.

What actually fixed it was that sentinel change: letting “it has never once succeeded” shout for itself, instead of waiting for someone to flip through the tables.

Three Things to Do Now

The sentinel is currently reporting 3 new problems, none of them agent-related:

1. git_hygiene — 10 merged-but-undeleted branches on this machine, the dirtiest being hym-manager (7)
2. service_down — on guqiuyue, hermes-gateway is not running
3. task_backlog — the P1 “dispatch dashboard” has sat queued for 6 days with nobody claiming it

On vex / nanzhu, agent-beat.sh still uses the same error-swallowing pattern (it hasn’t been exposed yet because it currently works). These three items plus hardening the script — I can do all of them directly.

The Friction in This Round

What really deserves to be remembered isn’t that I fixed two heartbeats. It’s that both of these failures were interrogated out of me by you, not reported by the system itself.

For several rounds before that I kept working on “how the table displays it” — but no matter how accurately you display it, someone still has to look.

Those three lines of continue in the sentinel were the reason they could stay silent for dozens of days, and yet I only went and read watch.go after your fourth round of pressing.