Text encodings

A String always holds UTF-8. The text module converts text to and from the bytes of an encoding (u8[]), for files, network protocols and C libraries that use another one:

# Text in other encodings: encode a String to bytes, decode bytes back.
import text;

main(): int {
    var greeting = "héllo 🙂";

    # UTF-8 is what a String holds; decode checks the bytes.
    var utf8 = text.Utf8.encode(greeting);
    assert(utf8.count == 11, "é takes two bytes, 🙂 four");
    assert(text.Utf8.decode(utf8).value() == greeting, "round trip");

    # UTF-16: two bytes per unit, little-endian unless asked otherwise.
    var utf16 = text.Utf16.encode(greeting);
    assert(utf16.count == 16, "🙂 takes a surrogate pair");
    assert(text.Utf16.decode(utf16).value() == greeting, "round trip");

    # ASCII has no é: strict gives none, lossy writes '?'.
    assert(text.Ascii.encode(greeting).is_empty(), "not ASCII");
    assert(text.Ascii.encode_lossy("é").count == 1, "one '?'");

    # Code page 437, as old zip archives name entries: é is 0x82, and every
    # byte decodes, so decode gives a String.
    var dos = text.CodePage437.encode("café").value();
    assert(dos.count == 4 && dos[3] == 0x82, "one byte each");
    assert(text.CodePage437.decode(dos) == "café", "round trip");
    assert(text.CodePage437.encode(greeting).is_empty(), "no 🙂 in code page 437");

    # Strict decoding rejects invalid bytes; lossy decoding replaces them.
    var bad = Array<u8>();
    bad.append(104);
    bad.append(255);
    assert(text.Utf8.decode(bad).is_empty(), "0xFF is never UTF-8");
    assert(text.Utf8.decode_lossy(bad) == "h\u{FFFD}", "replaced");
    return 0;
}

Type	Encoding
`text.Utf8`	UTF-8. `encode` copies the String’s bytes; `decode` checks them. `is_valid` checks bytes without decoding, and `find_invalid` gives the offset of a String’s first invalid byte. `code_point_at` reads one code point and `append_code_point` writes one.
`text.Utf16`	UTF-16, two bytes per code unit, little-endian by default (`encode(s, true)` and `decode(bytes, true)` for big-endian). A code point past U+FFFF takes a surrogate pair.
`text.Ascii`	ASCII: code points below 128, one byte each. `is_ascii` checks a String.
`text.CodePage437`	Code page 437, the IBM PC character set that old zip archives use for entry names: ASCII below 128, and one accented letter, Greek letter or box-drawing symbol for each byte above. Every byte decodes, so `decode` gives a String, not an Optional.

decode is strict: it gives Empty when the bytes are not valid in the encoding (a stray UTF-8 continuation byte, an unpaired UTF-16 surrogate, an odd UTF-16 byte count, an ASCII byte of 128 or more).
decode_lossy always succeeds: whatever is invalid becomes U+FFFD, the replacement character.
Ascii.encode gives Empty for text with a code point past 127, and CodePage437.encode for a character the code page lacks; encode_lossy writes ? for each instead.
No byte-order mark is written, and one that is read stays in the text.
io.File.read_string decodes with text.Utf8.decode, and an HTTP body’s text() with decode_lossy.

The reference lists every function.

Keyboard shortcuts

The Pepper Programming Language

Text encodings