I Ran an AI-Readability Checkup on My Own Project and the First Cut Landed on Me

I’m building a small tool called Scorecard that scores open-source projects across nine dimensions. When I added the ninth dimension, “AI readability,” this year, I casually ran it on my own project — and it turned out my own site was blocking AI crawlers, which I had no idea about.

What’s more embarrassing is today: while preparing this talk I checked again, and found there’s a second layer under the first one I fixed — what I thought was solved long ago had been there the whole time.

That process is what I want to talk about: what I saw, which traps I fell into, and a few questions I still haven’t figured out and would like to hear your thoughts on.

Why I Care About This in the First Place

Star counts tell you the result, not the reason. The change I noticed is that AI crawlers and coding assistants are becoming a new traffic entry point — users only click through because ChatGPT, Perplexity or Claude cited a project. The activity around robots.txt over the past few years is telling:

When	What	Impact
2023.08	OpenAI announces GPTBot, site owners start blocking	Content never makes it into AI search answers
2024	ClaudeBot / PerplexityBot / Google-Extended launch one after another	The blocklist keeps growing
2025	GitHub adds native AGENTS.md support, llmstxt.org launches	Repository-level “AI readability” gets a standard practice
2025+	Cloudflare’s managed robots.txt blocks AI crawlers by default	Many sites are blocked by default

This is my judgment, and it may be wrong: LLM citations are becoming a new traffic entry point. It might also not matter that much — I’d love to hear what you think.

So I Added a Ninth Dimension

The original eight dimensions were built from the angle of “would a stranger star this within 10 seconds.” For the ninth, AI readability, I set myself this constraint: every criterion must be objectively checkable — is the file there, what’s the status code — anything that can’t be judged isn’t counted against the project, and the score is normalized over the verifiable part.

AGENTS.md: What I Understand as “the First Sentence a Repository Says to AI”

Since 2025, GitHub has native support for AGENTS.md. Copilot / Codex / Claude Code read it first thing when entering a repo.

Here’s how I scored these items — to be clear, these nine dimensions and weights are all off the top of my head; I’m putting them out there mainly to hear how you’d rank them:

Criterion	Score I assigned	Notes
AGENTS.md in the repo root	+3	Behavior rules for AI assistants
llms.txt in the repo root	+2	Content index for LLMs
.github/copilot-instructions.md	+1.5	Custom Copilot instructions
Site’s robots.txt allows AI crawlers	+2	GPTBot / ClaudeBot etc.
llms.txt in the site root	+1.5	Content index for LLMs

Here’s How I Wrote Mine

• What this is (one sentence + a map of the key files)
• Common commands (build / test / start the server)
• Conventions (where to change things, how to change them, what’s forbidden)

# AGENTS.md — AI 助手工作守则

## 这是什么
Scorecard —— 开源项目九维度质检工具。
- server/audit.js：质检引擎（权威）
- src/components/Scorecard.vue：前端面板

## 常用命令
- bun run build    # 构建前端到 dist/
- bun run server   # 起后端（:54445）

## 约定
- 改维度必须同步 audit.js + check-dimensions + SKILL.md 三处
- 每条结论必须有证据，判不了写 manual + unverifiable

llms.txt: a Content Index for LLMs

The llmstxt.org spec is: put a plain-text index in the site root so an LLM can see at a glance “what’s here and what matters.” Mine looks like this:

# Scorecard

> 开源项目九维度质检。粘一个 GitHub URL，几秒钟拿到雷达图、
> 整改清单和可直接喂给 AI 的 Markdown 报告。

Key points:
- 输入：GitHub 仓库地址（owner/repo）
- 输出：0-10 总分 + 九维度雷达图 + 整改清单

Useful links:
- [在线使用](https://scorecard.webkubor.online)
- [源码仓库](https://github.com/webkubor/scorecard)

Here’s a trap I fell into: an SPA’s history fallback returns index.html for any path, so /llms.txt handed back a page of HTML instead of the index. I assumed dropping the file in was enough, but you actually have to register the route explicitly so the SPA doesn’t take over.

robots.txt: I Thought I Didn’t Need to Touch It, but It Was Blocked by Default

User agents of the mainstream AI crawlers:

UA	Whose	Purpose
GPTBot	OpenAI	Training and search
ClaudeBot / Claude-Web	Anthropic	Training and citation
PerplexityBot	Perplexity	AI search
Google-Extended	Google	AI training opt-out
CCBot	Common Crawl	Large-scale corpora

robots.txt is an opt-out protocol, no Disallow means allowed by default, so I always figured this wasn’t something I needed to worry about:

User-agent: *
Allow: /

Failure Layer One: Managed robots.txt

Cloudflare has a “managed robots.txt” feature that injects a row of Disallow rules into your site — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, nine in total, plus a complimentary Content-Signal: ai-train=no. My site had this feature on, and I had no idea what it was doing — the guy who built the checkup tool, first cut landing on himself.

Here’s how I found out:

curl -s https://你的域名/robots.txt | grep -iE "gptbot|claudebot|perplexity"

Failure Layer Two: I Turned It Off, and Crawlers Still Couldn’t Get In

Turned off the managed feature, curl showed Allow: /, and I thought I was done. Then just for good measure I hit my own site with GPTBot’s UA:

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.0" https://你的域名/
# 403

robots.txt is only a gentleman’s agreement; Cloudflare also has an ai_bots_protection that really blocks at the edge, returning 403 outright. These are two independent switches, and turning one off doesn’t mean the other is open — in my case, with robots.txt already saying Allow: /, I got a 403.

A few things that sent me in circles, for the record: the official docs only cover how to turn it on, not how to turn it off; some people tried overriding it with a Worker and couldn’t, because the injection happens at the edge; and the API has no endpoint called robots — the field is buried in bot_management under the name is_robots_txt_managed, and you have to dig through the API reference to find it. On by default + docs don’t say how to turn it off + buried under another product — “blocked by default without knowing it” isn’t anyone being careless, it’s the inevitable outcome of those three things stacking up.

So a self-check can’t stop at looking at robots.txt, you have to actually hit your site with a crawler’s UA.

A Reference Point, Scored with the Same Standard

Again: these scores come from my own off-the-cuff standard, so they’re only useful for relative comparison, not as a measure of project quality:

Project	AI readability	Highlights	Missing
sst/sst	6.5	Has AGENTS.md, site has llms.txt	Repo has no llms.txt
vitejs/vite	5	Has Copilot instructions, site has llms.txt	No AGENTS.md
denoland/deno	1.5	Has Copilot instructions	None of the rest
webkubor/scorecard	6.5 → 8.5	Added AGENTS.md / llms.txt	Site was once blocked by CF by default

Three Things I Plan to Do Next

1. Abstract the AGENTS.md template one more level; right now every project writes it by hand
2. Add auto-generation for llms.txt; maintained by hand, it will drift eventually
3. Make the robots.txt check a CI step, so it doesn’t get blocked by default again

If you’d like to take a look at your own site, two commands, both required:

# 第一层：robots.txt 里有没有被注入 Disallow
curl -s https://你的域名/robots.txt | grep -iE "gptbot|claudebot|perplexity"

# 第二层：爬虫实际进不进得来（403 就是被边缘拦了）
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.0" https://你的域名/

Running only the first one leaves a gap — that’s exactly how I missed it today.

A Few Questions I Haven’t Figured Out

These are ones I genuinely don’t have answers to, and I’d really like to hear from your experience:

1. Will the formats fragment: AGENTS.md, CLAUDE.md, .cursorrules, copilot-instructions each written separately — are you ultimately maintaining five copies of the same thing? Has anyone been through this, and how did you converge?
2. Is it worth doing for internal private repos: externally it’s a traffic benefit; what about internally? If the benefit is “newcomers and AI get up to speed faster,” how is that different from a well-written README?
3. Does allowing AI crawlers carry risk for commercial projects: content gets trained on, and traffic doesn’t necessarily come back. I’ve thought this through for open-source projects; for commercial sites I haven’t.