HHY Extension · Implemented in v1.1

HTML Extension and Crawler Framework

Use the official HTML extension and my-crawler for a real, auditable, bounded static-page collection task.

Why static-document collection comes first

HHY already has HTTP, timeout, retry, parallel, structured errors, and atomic file output. DOM parsing was the missing piece. The first version therefore does not turn Runtime into a browser; it adds a process extension with no network or file capabilities. HHY owns fetching and scheduling, while Lexbor parses untrusted HTML and evaluates CSS selectors.

Build and install the HTML extension

brew install jansson lexbor
make -C extensions/html
./build/hhy install ./extensions/html
./build/hhy list

Callable	Result	Purpose
html.text / text_all	String / List	Read normalized text from one or many nodes
html.attr / attr_all	String / List	Read one or many attributes
html.exists	Bool	Test whether a selector matches
html.extract	List<Map>	Parse once and project repeated records through a schema

The extension limits input to 768 KiB. Collections default to 1000 results with a hard limit of 10000, keeping protocol messages bounded. It does not return DOM handles: Protocol 1 transports only JSON-shaped values, so html.extract performs parsing and projection inside one extension call.

Initialize my-crawler

make
./practical-projects/my-crawler/init.sh
./practical-projects/my-crawler/self-test.sh
./practical-projects/my-crawler/run.sh

init.sh installs html into the project's own .hhy-extensions directory without touching the user-level extension home, then creates the Git-ignored output directory. self-test.sh verifies the complete HTTP → DOM → JSON path against a local fixture server; run.sh performs the real hhylang.dev documentation crawl.

{
  "seeds": ["https://hhylang.dev/zh/learn/cli-reference"],
  "parallelism": 2,
  "root_selector": "main article h2",
  "max_results": 100,
  "schema": {
    "title": { "selector": "", "value": "text" },
    "anchor": { "selector": "", "value": "attr", "name": "id" }
  }
}

Real run and explicit boundaries

Actual run

$./practical-projects/my-crawler/run.sh

HHY Collector Framework HHY Documentation Crawler
Pages 1 / 1 Records 6 Failures 0
Records practical-projects/my-crawler/output/records.json
Report practical-projects/my-crawler/output/report.json

Layer	Responsibility
Crawler	Seed deduplication, bounded parallelism, timeout/retry, per-page failure archive
HTML extension	Lexbor DOM, CSS selectors, text/attribute/schema extraction
Output	Three atomic JSON files: records, report, and failures
Current boundary	Buffered response bodies; no JS rendering, browser automation, or unbounded streams

Read the complete my-crawler sourceIncludes bilingual documentation, a real task configuration, initialization, a local fixture server, and deterministic end-to-end verification.