werkator/docs/prs/2026-09-08-PR#25-robots-txt.md

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

The Problem

On 2026-09-05 and 2026-09-06 the mih09 webspace produced 4 GB and 7 GB of Internet traffic in one day each, against a baseline of about 50 MB. The Apache logs showed two AI crawlers (GPTBot, ClaudeBot) walking the per-commit views of the Gitea next to Werkator; that part was fixed on the host. The same crawlers, plus Amazonbot and Googlebot, also walk Werkator itself: on vm4006 they downloaded 1.9 GB of build artifacts (JaCoCo reports, dependency license trees, test reports) in a week. Artifacts form an unbounded URL space — every build adds a full report tree — so a crawler never finishes and the traffic returns with every build. Werkator answered /robots.txt with a 404, so even polite crawlers had nothing to obey. A robots.txt placed by hand in the webspace's docroot (as done on mih09) is lost when instance-start regenerates the .htaccess, and the managed nginx of vm4006 has no docroot at all.

Non-Goals

• Blocking crawlers that ignore robots.txt — that stays with the proxy in front of Werkator (a user-agent block in the .htaccess), because Werkator cannot refuse cheaper than the proxy can.
• Access control on artifacts — anonymous downloads by people and scripts (the werkdock binary is fetched from a branch URL) stay open.
• Changes to the Gitea next to Werkator.

The Scenarios

Feature: Werkator serves a robots.txt of its own

Background

• Artifact routes: /artifacts/<key>/…, /branches/<branch>/…, /builds/<key> and their /repos/<name>/… forms.
• Werkator serves static files from src/main/resources/static/ at the root, like werkator.js.

Scenario#25.01: Crawlers are steered off artifacts and branch files

So that a polite crawler indexes the build pages but never walks the report trees.

• Given a running Werkator behind any reverse proxy
• When a crawler requests /robots.txt
• Then it receives a robots.txt that disallows the artifact, branch-file, build and API routes in both the unscoped and the /repos/<name>/ form
  • and the pages themselves remain allowed.

Verified by

• UiControllerTest — robots.txt is served by Werkator itself

The Solution

src/main/resources/static/robots.txt is a static resource; Spring serves it at /robots.txt like the other static files. Serving it from Werkator itself, rather than generating it with init --systemd as first intended, covers every proxy at once: the Apache .htaccess of a managed webspace proxies everything to Werkator anyway, and the managed nginx container has no docroot to place a file in. Neither the generated .htaccess nor the nginx template nor tools/remote needs a change, and a regenerated .htaccess cannot lose the file. The Crawl-delay is a hint for the crawlers that honour it.

Additional Changes

• docs/deployment.md names the served robots.txt and where a block of robots-ignoring crawlers belongs.

Follow-up PRs

• Optionally a user-agent block for known robots-ignoring crawlers in the generated .htaccess, once the hand-maintained one on mih09 has proven its list.