When the user reported it, they brought exactly one screenshot: all four thumbnails under “Recent generations” in the image workspace were broken, and even the preview of an uploaded reference image was broken. The one-line description was “a lot of images won’t display properly today.”
Meanwhile, when I pulled the same batch straight from the origin on the server, 18 samples, 18 200s.
That is what eats the most time in this class of outage: it looks exactly like our own bug, and it isn’t.
First, Separate “Upstream” from “Us”
Image addresses in the system carry two identities:
库里存的永远是源站地址 https://img.webkubor.online/...
│ 镜像回源（按需拉取 + 落盘）
▼
渲染时才派生出的加速地址 https://<bucket>.s3.bitiful.net/...
The origin is stored in the database and swapped for the domestic accelerated domain at display time, so the database isn’t tied to a CDN — the design is sound. The problem is that the swap was a plain string replacement: once the accelerated domain is unavailable, the browser just shows a broken image and never goes back to the origin.
So the first step isn’t reading code, it’s hitting the same objects on both domains:
| Object | Origin | Accelerated domain |
|---|---|---|
| 3 generated today | 200 / 200 / 200 | 424 (origin fetch failed) / hung / hung |
| 3 old images from early August | 200 ×3 | 404 ×3 |
| 4 cached old images | 200 ×4 | 200 ×4 |
Not a single image was broken at the origin. That narrows the scope from “the entire image pipeline” down to “that one derived address.”
Misjudgment One: I Thought It Was a Key Rotation
My first hypothesis was tidy: the origin is R2, the acceleration is someone else’s mirror bucket, and the mirror needs R2 credentials to fetch from origin;
the R2 credential in the vault was annotated r2-s3-rclone-20260901, and the last object that successfully landed in the accelerated bucket was dated 8-31.
“Key rotated on 9-01, never synced to the mirror” — the timeline fit perfectly.
Then two things killed it at once:
- In the account’s API token list, the most recent change was exactly that 9-01 entry, and nothing has been rotated since;
- More importantly, “it was fine yesterday.” If it had broken on 9-01, the user would have reported it on 9-01, not waited until today.
A timeline is the cheapest falsification tool there is. Once I had the two timestamps — last success and first failure — that hypothesis was dead.
Misjudgment Two: I Thought It Was Just This Bucket’s Config
Once the timestamps lined up, I ran a control: another project under the same account also has an accelerated bucket.
One key that doesn’t exist:
- That bucket: 404 in 0.05 seconds (clean and quick)
- Our bucket: hangs for 12 seconds (the connection is fine, the response just never comes)
“Only our bucket hangs → our bucket’s origin-fetch config is wrong.” That reasoning looks fine too — until I realized: “doesn’t exist” and “exists but isn’t cached” don’t take the same path. A path with an origin-fetch rule goes back to origin (and hangs); a path with no rule returns 404 directly (instantly). I had read the behavior of two different code paths as a difference between buckets.
The Decisive Experiment: Try Another Bucket
Pinning the blame took exactly one step: put a brand-new object on the origin, then fetch it through another project’s bucket.
写入 R2：hym/__probe_<ts>.txt
源站取： 200 ← 源站有货
hym 桶取： 000 / 25 秒超时 ← 也是挂的
Different bucket, different project, origin fetch still hangs. Conclusion landed: the account-level origin-fetch capability is broken, not our bucket’s configuration. (The probe object was deleted right after; I double-checked.)
That step was fast because it eliminates the “our config” variable: same account, same mechanism, another bucket I could not possibly have misconfigured — and it doesn’t work either, so it isn’t a configuration problem.
Why “It Was Fine Yesterday”
The accelerated responses carry cache-control: max-age=604800 — 7 days.
Images already sitting in edge or browser cache keep rendering normally; only objects that aren’t in cache and must go back to origin are exposed. That’s why the user saw a sudden burst of broken images rather than “the whole site instantly broken”; and why things were fine at 22:18 last night and dead at 9:07 this morning.
Partial failure looks the most like your own bug: half good, half bad makes you want to blame “some piece of code.” Its fingerprint is actually quite specific — the on-demand path is dead, the already-stored path is fine.
Is There Any Suspicion on Our Side: Ruled Out, One by One
| Suspicion | How I ruled it out | Result |
|---|---|---|
| Code changes | Zero commits in the repo during the outage window (both commits that day came after it started, and the first deploy was 3 hours 45 minutes later) | Ruled out |
| Cloud account config | Pulled every change in the window from the audit log: all certificates and DNS for unrelated sites | Ruled out |
| Origin credentials | Account token list: no changes in the window; current credentials tested and valid | Ruled out |
| The origin itself | 18 samples plus one freshly written object, all 200 | Ruled out |
Only one direction is left: the acceleration provider’s account side. That isn’t in our hands — but it shouldn’t be something we only learn about when a user reports it.
The Real Problem: The User Found It, Not the Alerts
One look at our health checks: every endpoint returns 200, product library, asset library and health check all green. All endpoints green, all images shattered — the checks look at status codes and response shapes; images aren’t in their field of view at all.
Already fixed: four probes per run, and neither class of object is optional.
| What we probe | Why |
|---|---|
| Stable object (brand logo) × both domains | Verifies the image host domains are still alive (DNS / certificate / edge / storage) |
| The most recent generated image (fetched live from the database) × both domains | Verifies origin fetch still works — probing only stable objects would have missed this outage entirely |
The verdict is two-tiered: failing to fetch from origin triggers a Feishu alert (users see broken images; that’s a real outage); the accelerated domain only reports status without alerting (when it’s down, functionality is unaffected, the frontend has a fallback), but it rides along in the alert card — so when it recovers you can see at a glance that “we can turn it back on.”
The Patch Had a Bug of Its Own
I shipped the first version of the check and immediately hit the production endpoint: empty response. The same handler came back in 3.5 seconds locally.
The difference is the egress path: that “hanging accelerated request” never gets a response when it leaves through Cloudflare,
and my check code had no timeout whatsoever — one hung request stalled the whole endpoint, so the wget line in CI naturally got nothing.
The check endpoint itself became a silent failure, which is exactly the class of problem it exists to catch. Every request now carries a 6-second timeout, and the lesson went straight into the comments: a check endpoint must have its own timeout, otherwise it turns “the probed object hangs” into “the probe hangs.”
Landing It: What Changed
- No longer depending on that accelerated domain: nothing is injected at build time, images go straight to the origin. Loading is slower in mainland China (measured 6 seconds vs 0.4 seconds for the same image) — trading speed for “it actually shows up” is worth it.
- Added an origin fallback: if the accelerated address fails to fetch, or hasn’t finished loading in 2.5 seconds, automatically retry once against the origin.
Using the capture-phase resource error event (these errors don’t bubble) plus a timeout check, one place covers all 27 display points,
with no changes to 15 view files. Measured end to end: a hanging image goes from
naturalWidth0 to 1104. - Added images to the health check: twice a day, morning and evening; origin problems push straight to Feishu.
- Recovery criteria written into the check: wait for “accelerated · latest generation” to go green again, then flip the build switch back on.
What’s Still Unresolved, and What Isn’t in Our Hands
The origin-fetch failure on the acceleration provider’s account side is still there. Their console API only accepts a logged-in session (we tried 6 different auth schemes, all of them “please log in first”), so balance, quota and account status are all unqueryable — either a human logs in and looks, or we keep waiting.
It doesn’t affect whether images display, only how fast. That boundary needs to be stated clearly, or people will assume “it’s still not fixed.”
Three Things Worth Reusing
- Prove it isn’t you first, then talk about fixing it. Hitting the same objects on two domains / two environments is far faster than reading code.
- Try the same class of object somewhere you couldn’t possibly have misconfigured. It’s the cheapest blame-assignment experiment — with the variable eliminated, the conclusion isn’t arguable.
- Read the fingerprint of a partial failure. Stored objects fine, on-demand dead means the problem is on the on-demand path; chasing “did we write some code wrong” will take you the long way around.
One more, about process: the reason this went so fast is that the origin’s credentials were in the vault; whereas that accelerated bucket, nobody can get into — because after it was created, its credentials were never registered anywhere. “In use but not in the ledger” is itself an accelerant for the next outage.
