Clean Text and Match Patterns

Today we will create an alert file from a small application log. The input deliberately includes leading spaces, mixed case, and sentences that mention errors without being error records. Our goal is to keep real warning and error severity lines, not every sentence containing similar letters. Run the supplied script from examples and keep the input log visible beside the generated alerts file as we explain each step.

hhy run clean-log.hhy

Run the complete example from its examples directory.

String functions return new values. Trim removes surrounding whitespace, replace changes the exact text we request, and lower converts letter case. The original binding still refers to the original text. This pipeline is a useful way to see how one value moves through several transformations. We are demonstrating normalization here, so changing ERROR to WARN is intentional. The actual alert extractor keeps the original severity unchanged.

let original = "  ERROR: timeout  "
let cleaned = original |> trim
    |> replace("ERROR", "WARN") |> lower
print(cleaned)

warn: timeout

Run the complete example from its examples directory.

Text length can mean different things. In HHY, length counts Unicode code points, while byte_length counts encoded bytes. These two Chinese characters therefore produce two and six. A code point count is still not a universal measure of visible screen characters, because some displayed symbols combine multiple code points. Use the measure that fits your task, and avoid truncating encoded text by treating a byte position as a character index.

print(length("你好"))
print(byte_length("你好"))

2
6

Run the complete example from its examples directory.

Read_lines gives us one line at a time. The map callback trims each line before matching, which matters because our pattern will check the beginning of the text. If we skipped normalization, the leading spaces on the warning record would prevent that anchor from matching. Keep the order of operations deliberate: first establish the representation you want to inspect, then apply the rule to that normalized representation.

read_lines(path("fixtures/sample.log"))
    |> map { line -> trim(line) }

Run the complete example from its examples directory.

A regex literal places the pattern between slashes. The caret anchors this pattern at the beginning. The alternatives allow ERROR or WARN, and the word boundary prevents warning from being accepted as the shorter WARN token. The i flag ignores case. Regex_match returns a Boolean, making it suitable for a where predicate. Read the pattern aloud as a rule before applying it to an entire input stream.

regex_match("WARN cache slow", /^(ERROR|WARN)\b/i)

Run the complete example from its examples directory.

Where retains lines whose predicate is true. It does not rewrite their content, so our trimmed warning and error messages pass through intact. Save_lines then consumes the result and writes the alert file. The fixture should produce exactly two lines. The informational sentence mentioning errors is excluded, and so is the line beginning with warning. Those near misses are useful evidence that our rule is more precise than substring searching.

read_lines(path("fixtures/sample.log"))
    |> map { line -> trim(line) }
    |> where { line ->
        regex_match(line, /^(ERROR|WARN)\b/i)
    }
    |> save_lines(path("out/alerts.log"),
        { create_parents: true })

WARN cache slow
ERROR request timeout

Run the complete example from its examples directory.

Regex_captures is useful when you need details from a match, such as a captured message. But when nothing matches, it returns null. Check that possibility before accessing capture fields. This demonstration intentionally searches an informational line for an error pattern, so the comparison prints true. Regex_match is simpler when you need only yes or no; choose captures when extracting structured values is actually part of the task.

let miss = regex_captures("INFO ready", /ERROR (.*)/)
print(miss == null)

true

Run the complete example from its examples directory.

A broad search for error or warn anywhere in a line would accept both of these distractors. That may be right for a full text search, but it is wrong for our severity extractor. Requirements decide the pattern. Do not make a regex more complicated just to make it look powerful. Start with representative positive and negative examples, then verify that your anchors and boundaries express the intended log format.

regex_match("info no errors here", /error/i)
regex_match("warning is not a token", /warn/i)

Run the complete example from its examples directory.

Add a lowercase error record to the fixture and an informational record that mentions warning. Predict which one will enter alerts before running the script. Then extend the allowed severity alternatives to include FATAL, and add a corresponding test line. Keep one near miss for every new rule. You now have a repeatable method for cleaning text, choosing a match predicate, and checking that unwanted records stay out of the output.

# Add: error lowercase test
# Add: INFO warning count zero
hhy run clean-log.hhy

Run the complete example from its examples directory.