Serve a robots.txt that keeps crawlers off build artifacts #25

2026-09-08T09:34:06+02:00

mi commented

2026-09-08 09:34:06 +02:00

Build artifacts are an unbounded URL space; AI and search crawlers walked
them in full on vm4006 (1.9 GB in a week) and answered 404 on /robots.txt.
Served by Werkator itself so every proxy in front of it delivers it without
a host-level file that a regenerated .htaccess could lose.

Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com

Build artifacts are an unbounded URL space; AI and search crawlers walked them in full on vm4006 (1.9 GB in a week) and answered 404 on /robots.txt. Served by Werkator itself so every proxy in front of it delivers it without a host-level file that a regenerated .htaccess could lose. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mi added 1 commit 2026-09-08 09:34:06 +02:00

server: serve a robots.txt that keeps crawlers off artifacts and branch files 67159f9292

Build artifacts are an unbounded URL space; AI and search crawlers walked
them in full on vm4006 (1.9 GB in a week) and answered 404 on /robots.txt.
Served by Werkator itself so every proxy in front of it delivers it without
a host-level file that a regenerated .htaccess could lose.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mi added 1 commit 2026-09-08 09:34:21 +02:00

docs(prs): PR#25 has its number 16c6db3128

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

mi marked the pull request as ready for review 2026-09-08 10:09:44 +02:00

mi closed this pull request

2026-09-08 10:09:56 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mi/werkator#25