Project · Self-tested

Project: DataFlow ETL

Synchronize CSV, JSON-directory, and HTTP API data through cleaning, filtering, concurrent enrichment, grouping, and JSON/CSV outputs.

A complete synchronization pipeline

DataFlow ETL is written entirely in HHY v1.0 and verified end to end. It reads customer CSV and an event JSON directory, normalizes names and email addresses, filters inactive and low-spend customers, calls a profile API concurrently, aggregates departments with group_by, and atomically writes JSON and CSV outputs.

Stage	Implementation
Ingest	read_lines + parse_csv; files + parse_json
Clean	trim, lower, to_int, and structured Maps
Enrich	parallel(4) + http.get + timeout + retry
Aggregate	where, sort_by, group_by, and sum
Output	encode_json/save_text and encode_csv/save_lines

View the complete DataFlow ETL source on GitHub ↗Includes HHY modules, CSV/JSON fixtures, profile API, report assertions, and one-command self-test.

Real layout and data flow

DataFlow ETL project tree — The real project layout: entry point, four HHY modules, CSV/JSON fixtures, configuration, and test utilities.

customers.csv + events/*.json + HTTP profiles
                    ↓
            parse / trim / lower
                    ↓
          active + minimum spend filter
                    ↓
          parallel HTTP enrichment
                    ↓
          group_by department + sum
                    ↓
              report.json + customers.csv

Actual self-test result

cd hhy-vm
sh dataflow-etl/self-test.sh

Actual DataFlow ETL end-to-end result — Actual run: three qualified customers, two event files, two department summaries, and all HTTP enrichment plus JSON/CSV assertions passed.

Run your own synchronization

Copy config/test.json, replace the project name, API base, and minimum-spend threshold, then provide customers.csv and events/*.json. An individual HTTP failure becomes a structured error, sets report ok=false, and returns exit code 1.

hhy run dataflow-etl/etl.hhy \
  ./input \
  ./config.json \
  ./output/report.json \
  ./output/customers.csv