56 lines
3.4 KiB
Markdown
56 lines
3.4 KiB
Markdown
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
|
|
|
## The Problem
|
|
|
|
On 2026-09-05 and 2026-09-06 the mih09 webspace produced 4 GB and 7 GB of Internet traffic in one day each, against a baseline of about 50 MB.
|
|
The Apache logs showed two AI crawlers (GPTBot, ClaudeBot) walking the per-commit views of the Gitea next to Werkator; that part was fixed on the host.
|
|
The same crawlers, plus Amazonbot and Googlebot, also walk Werkator itself: on vm4006 they downloaded 1.9 GB of build artifacts (JaCoCo reports, dependency license trees, test reports) in a week.
|
|
Artifacts form an unbounded URL space — every build adds a full report tree — so a crawler never finishes and the traffic returns with every build.
|
|
Werkator answered `/robots.txt` with a 404, so even polite crawlers had nothing to obey.
|
|
A `robots.txt` placed by hand in the webspace's docroot (as done on mih09) is lost when `instance-start` regenerates the `.htaccess`, and the managed nginx of vm4006 has no docroot at all.
|
|
|
|
## Non-Goals
|
|
|
|
- Blocking crawlers that ignore `robots.txt` — that stays with the proxy in front of Werkator (a user-agent block in the `.htaccess`), because Werkator cannot refuse cheaper than the proxy can.
|
|
- Access control on artifacts — anonymous downloads by people and scripts (the werkdock binary is fetched from a branch URL) stay open.
|
|
- Changes to the Gitea next to Werkator.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: Werkator serves a robots.txt of its own
|
|
|
|
#### Background
|
|
|
|
- Artifact routes: `/artifacts/<key>/…`, `/branches/<branch>/…`, `/builds/<key>` and their `/repos/<name>/…` forms.
|
|
- Werkator serves static files from `src/main/resources/static/` at the root, like `werkator.js`.
|
|
|
|
#### Scenario#25.01: Crawlers are steered off artifacts and branch files
|
|
|
|
So that a polite crawler indexes the build pages but never walks the report trees.
|
|
|
|
- **Given** a running Werkator behind any reverse proxy
|
|
- **When** a crawler requests `/robots.txt`
|
|
- **Then** it receives a `robots.txt` that disallows the artifact, branch-file, build and API routes in both the unscoped and the `/repos/<name>/` form
|
|
- **and** the pages themselves remain allowed.
|
|
|
|
##### Verified by
|
|
|
|
- [UiControllerTest — robots.txt is served by Werkator itself](../../src/test/kotlin/de/hoennig/werkator/server/UiControllerTest.kt)
|
|
|
|
## The Solution
|
|
|
|
`src/main/resources/static/robots.txt` is a static resource; Spring serves it at `/robots.txt` like the other static files.
|
|
Serving it from Werkator itself, rather than generating it with `init --systemd` as first intended, covers every proxy at once: the Apache `.htaccess` of a managed webspace proxies everything to Werkator anyway, and the managed nginx container has no docroot to place a file in.
|
|
Neither the generated `.htaccess` nor the nginx template nor `tools/remote` needs a change, and a regenerated `.htaccess` cannot lose the file.
|
|
The `Crawl-delay` is a hint for the crawlers that honour it.
|
|
|
|
## Additional Changes
|
|
|
|
- `docs/deployment.md` names the served `robots.txt` and where a block of robots-ignoring crawlers belongs.
|
|
|
|
## Follow-up PRs
|
|
|
|
- Optionally a user-agent block for known robots-ignoring crawlers in the generated `.htaccess`, once the hand-maintained one on mih09 has proven its list.
|