Project · Self-tested
Project: SiteGraph Auditor
Recursively inventory a documentation site, build a normalized link graph, and enforce metadata, failure, and security gates.
A quality gate that uses the safe spider end to end
SiteGraph Auditor builds on HHY v1.1.4 and the safe recursive my-crawler engine. Starting from seeds, it discovers pages by depth and writes an inventory, normalized graph, report, and failures. A stable exit status blocks sites with missing metadata or crawl failures.
| New capability | Use in this project |
|---|---|
| URL normalization | Unify relative links, dot segments, fragments, host case, and default ports |
| Link discovery | main a[href] continuously feeds the next frontier |
| Frontier | Concurrent depth batches retain page, depth, and source context |
| Hard boundaries | Domain, path, depth, pages, frontier, and links |
| Fingerprint deduplication | Deduplicate before frontier admission; count duplicate graph edges separately |
| SSRF | Production configuration rejects private resolved socket addresses across DNS and redirects |
Healthy and risky scenarios
make
./practical-projects/sitegraph-auditor/self-test.sh$./practical-projects/sitegraph-auditor/self-test.sh
SiteGraph Auditor healthy Pages 4 / 4 Edges 5 Duplicates 3 Rejected 1 Findings 0 SiteGraph Auditor risky Pages 1 / 2 Edges 2 Duplicates 0 Rejected 1 Findings 3 SiteGraph Auditor self-test passed
The healthy site has four levels, relative URLs, dot segments, fragment duplicates, and an external reference. The risky site has missing description/canonical metadata, a 404, and an out-of-scope path. A final request proves safe mode rejects loopback.
Outputs and boundary
| Output | Content |
|---|---|
| inventory.json | title, description, canonical, heading, and source_url |
| graph.json | source, raw href, normalized target, fingerprint, allowed state, and rejection reason |
| report.json | Pages, edges, duplicates, rejections, limits, errors, warnings, and findings |
| failures.json | URL, depth, and stable error |