Project: Build a Log Alert Report

This project brings file discovery, regular expressions, streams, and parallel work together. We have two small log files containing ordinary information messages, warnings, and errors. Our goal is a report containing only the warning and error lines, with the source filename attached to each one. The final output should be deterministic and easy to inspect. We will build the pipeline around that result, then check both the successful run and a clear command line usage error.

api.log: WARN slow query
api.log: ERROR request failed
worker.log: WARN queue full

Actual report.txt from the included fixtures.

The script accepts two arguments: a directory containing logs and the destination report. Keeping those choices outside the source code makes the same script reusable. Run the command from the examples directory so the fixture path resolves as shown. The input is deliberately small, and the script does not include a minimum file size filter. That means every supplied log participates in the lesson, and you can verify every selected and rejected line without scrolling through a large production dataset.

hhy run log-alerts.hhy fixtures report.txt

Two arguments: input directory and output file.

Before accessing either argument, check that both were provided. If the count is wrong, print a concise usage message to standard error and exit with code three. This is a small decision with a large effect on usability. Someone who runs the script incorrectly gets instructions instead of an index error. It also makes automation easier, because a caller can distinguish an invalid invocation from successful report generation without guessing from the presence or absence of an output file.

if length(args) != 2 {
    print_error("usage: log-alerts.hhy <log-dir> <output>")
    exit(3)
}

Run without arguments to see exit code 3.

Convert the first argument to a Path, then discover log files recursively. Filesystem traversal order is not the report contract, so we sort by filename before processing. The explicit ascending option is part of the validated call form. For these fixtures, the filenames are unique, which makes the resulting order straightforward. If your real directory has repeated basenames in different subdirectories, include a relative path in the label and ordering strategy so readers can distinguish those sources.

path(args[0]) |> files("**/*.log")
    |> sort_by({ order: "asc" }) { file -> file.path.name }

The fixture filenames are unique.

Each worker reads one file as a line stream. The regular expression matches either ERROR or WARN, and map prefixes the selected line with its filename. We use explicit string concatenation here, matching the tested examples. Collect inside the worker returns a finite list, rather than trying to send an open file stream back across the worker boundary. The pattern is intentionally simple and case sensitive. It can match those words anywhere, so a structured log format may need a stricter expression.

    |> parallel(2) { file ->
        read_lines(file.path)
            |> where { line -> regex_match(line, /ERROR|WARN/) }
            |> map { line -> file.path.name + ": " + line }
            |> collect
    }

Each worker returns a finite List of report lines.

Parallel returns one result for every input file. Because each result is a list of matching lines, the next stage must flatten those lists into a single stream. Flat map converts each list back into a Stream and emits its lines. Save lines then consumes that final stream and writes the destination. Forgetting this flattening step is a common structural mistake: a list of lists is not the same data shape as a stream of text lines.

    |> flat_map { lines -> lines |> stream }
    |> save_lines(path(args[1]))
print("report saved")

One List per file becomes one stream of lines.

Run the project and open the report. The message report saved is useful feedback, but it is not the whole validation. Compare the file with the expected three lines: two from API and one from worker. Confirm that the information messages are absent and that the file ordering is stable. Then run without arguments and observe the usage failure. An unreadable file is not silently ignored by this example; the unhandled error should stop the pipeline instead of presenting a partial report as complete.

hhy run log-alerts.hhy fixtures report.txt
cat report.txt

Expect exactly three report lines.

Add three lines to a fixture: an information message, an uppercase warning, and a lowercase warning. Predict which will match before running the script again. Then decide whether your intended log format needs case insensitive matching or a more precise severity field. A useful report begins with a clear definition of what counts as an alert. You now have a complete small utility with a command contract, bounded workers, explicit data shapes, and output that can be verified independently of the terminal message.

Exercise: add INFO, WARN, and lowercase warn lines.

Predict the report before running again.