HHY Extension · Current version v1.7.1
HTML Extension and Crawler Framework
Combine the official HTML extension with a safe frontier for URL normalization, discovery, recursive deduplication, and bounded static crawling.
What the HTML extension is
The official html 0.2.0 package is a process extension with no network or filesystem effects. Lexbor parses untrusted HTML, evaluates CSS selectors, and extract_report returns batch rows with explicit truncation metadata.
Build and install the HTML extension
brew install jansson lexbor
make -C extensions/html
./build/hhy install ./extensions/html
./build/hhy list| Complete signature | Purpose |
|---|---|
| html.text(String html, String selector, Map?) -> String? | Read normalized text from the first matching node |
| html.text_all(String html, String selector, Map?) -> List<String> | Read text from every matching node |
| html.attr(String html, String selector, String name, Map?) -> String? | Read an attribute from the first matching node |
| html.attr_all(String html, String selector, String name, Map?) -> List<String> | Read an attribute from every matching node |
| html.exists(String html, String selector) -> Bool | Test whether a selector matches |
| html.extract(String html, String selector, Map schema, Map?) -> List<Map> | Parse once and project repeated records through a schema |
| Option | Callables | Behavior |
|---|---|---|
| trim: Bool | text, text_all, attr, attr_all | Trim surrounding whitespace; enabled by default |
| max_results: Int | text_all, attr_all, extract | Defaults to 1000; hard limit 10000 |
The extension limits input to 768 KiB, keeping protocol messages bounded. extract schema fields use { selector, value: "text" } or { selector, value: "attr", name }; an empty selector reads the current root. It does not return DOM handles: Protocol 1 transports only JSON-shaped values, so html.extract performs parsing and projection inside one extension call.
From HTML extraction to a safe crawler
The static spider composes separate responsibilities: Runtime url_resolve normalizes relative references; the HTML extension discovers links; a depth frontier checks domains, paths, page and queue budgets, and fingerprints before admission; the HTTP layer blocks SSRF at the resolved connection address.
| Capability | Current implementation |
|---|---|
| url_resolve(url, base?) | Returns url, scheme, host, port, path, query, and a stable fingerprint |
| Link discovery | follow_selector extracts href values and feeds the next frontier |
| Frontier and deduplication | Depth batches with bounded concurrency and pre-admission fingerprint deduplication |
| Hard boundaries | Domains, path prefixes, depth, pages, frontier size, and links per page |
| SSRF protection | Reject loopback, private, and link-local socket addresses, including redirects |
Scope and explicit boundaries
The crawler supports an in-memory or atomically checkpointed frontier, strict configuration-matched resume, and send_to response files. An optional Playwright renderer executes JavaScript separately from the side-effect-free Lexbor extension and applies domain plus DNS private-network checks to documents, redirects, and subresources.