Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Iteration

A for loop visits each element of a collection in turn. You can loop over an Array, over the code points of a String, over a range of numbers (0..count), and over any type of your own that follows the iterator protocol.

Looping over an array

for (var x in source) { ... } runs the body once per element, with x bound to the element:

struct Item {
    int value;
    construct(this.value = 0) {}
}

make_items(): Array<Item> {
    var items = Array<Item>();
    items.append(Item(20));
    items.append(Item(22));
    return items;
}

main(): int {
    var nums = Array<int>();
    nums.append(1);
    nums.append(2);
    nums.append(39);

    # The basic loop: x takes each element in order.
    var total = 0;
    for (var x in nums) {
        total = total + x;
    }
    assert(total == 42, "sum of the elements");

    # The loop variable may be typed, and the parentheses are optional.
    long wide = 0;
    for (long x in nums) {
        wide = wide + x;
    }
    for var x: int in nums {
        wide = wide - x;
    }
    assert(wide == 0, "typed loop variables");

    # break and continue work as in a while loop.
    var odd = 0;
    for (var x in nums) {
        if (x % 2 == 0) {
            continue;
        }
        if (x > 10) {
            break;
        }
        odd = odd + x;
    }
    assert(odd == 1, "only 1 is odd and below 10");

    # A struct element is shared: changing a field changes the element,
    # but rebinding the loop variable does not touch the array.
    var items = make_items();
    for (var item in items) {
        item.value = item.value * 10;
        item = Item(99);
    }
    assert(items[0].value == 200, "first element was updated");
    assert(items[1].value == 220, "second element was updated");

    # Iterating a temporary keeps it alive for the whole loop.
    var sum = 0;
    for (var item in make_items()) {
        sum = sum + item.value;
    }
    assert(sum == 42, "temporary array stays alive");

    # The loop is shorthand for driving the iterator yourself.
    var it = nums.iterate();
    var manual = 0;
    while (it.next()) {
        manual = manual + it.current();
    }
    assert(manual == 42, "the same elements, by hand");

    # Modifying the array after the loop is fine.
    nums.append(100);
    assert(nums.count == 4, "appended after the loop");
    return 0;
}

The example shows each part of the loop:

  • The loop variable can be declared with var or with a type: for (var x in nums), for (long x in nums), and for (var x: int in nums) all work.
  • The parentheses are optional, as with if and while: for var x in nums { ... }. The braces are required.
  • break and continue work as they do in a while loop.
  • The loop variable is a copy, like any var. Assigning a new value to it does not change the collection. For a struct element, though, the copy is another reference to the same object (see Memory and reference counting), so item.value = ... does change the element.
  • A temporary source stays alive. The loop holds a reference to what it iterates, so for (var item in make_items()) is safe.

Looping over a string

A String yields its code points one at a time. Each one is a String holding a single code point, the same value that indexing gives you (word[i]):

main(): int {
    var word = "héllo ✓";

    # A String yields one code point at a time, as a one-code-point String.
    var count = 0;
    var copy = "";
    for (var c in word) {
        assert(c.length == 1, "each piece is one code point");
        copy.append(c);
        count = count + 1;
    }
    assert(count == 7, "7 code points");
    assert(copy == word, "the pieces rebuild the string");

    # Each piece is the same value as indexing the string.
    var i = 0;
    for (var c in word) {
        assert(c == word[i], "same as word[i]");
        i = i + 1;
    }

    # An empty string runs the body zero times.
    for (var c in "") {
        assert(false, "never runs");
    }
    return 0;
}

Looping over a range

a..b is the numbers from a up to, but not including, b, and a..>b includes b. Looping over one is how you count:

main(): int {
    # `a..b` counts from a up to, but not including, b.
    var total = 0;
    for (var i in 0..5) {
        total += i;
    }
    assert(total == 10, "0 + 1 + 2 + 3 + 4");

    # `a..>b` includes b.
    var product = 1;
    for (var i in 1..>5) {
        product *= i;
    }
    assert(product == 120, "5 factorial");

    # The bounds give the type: `count` is a ulong, so i is too.
    var names = ["Ada", "Grace", "Edsger"];
    ulong letters = 0;
    for (var i in 0..names.count) {
        letters += names[i].size;
    }
    assert(letters == 14, "letters in every name");

    # A range that starts past its end is empty.
    for (var i in 3..1) {
        assert(false, "never runs");
    }

    # A range is a value: test membership, or use the iterator adapters.
    var digits = 0..10;
    assert(digits.contains(9) && !digits.contains(10), "half-open");
    var evens = 0;
    for (var even in (1..>10).filter((n) => n % 2 == 0)) {
        evens += even;
    }
    assert(evens == 30, "2 + 4 + 6 + 8 + 10");
    return 0;
}

  • The bounds give the type. A range’s element type is inferred from its bounds like a generic call’s type argument, so 0..names.count counts in ulong. The bounds must be integers; anything else is an error at the operator:

    main.pep:2:22: error: range bounds must be integers, not 'String'
    2 |     for (var c in "a".."z") {
      |                      ^~

  • Precedence. A range binds more loosely than arithmetic and shifts, and more tightly than comparisons, so 0..n + 1 is 0..(n + 1). Two range operators in a row (1..2..3) are an error; add parentheses. ... is not a range operator.

  • An empty range runs no iterations. A range whose start is past its end (3..1, 5..>4) is empty, and a..a is too. A closed range ending at its type’s largest value (250..>top with a byte top of 255) stops there without overflowing.

  • A range is a value, a Range<T>. It has contains(x) and the is_empty property, and it is an Iterator<T>, so it has the adapters (map, filter, enumerate, zip) and to_array(). Looping over one allocates nothing; the optimizer turns it into a plain counting loop.

The iterator protocol

The for loop is not tied to Array and String. It works with any type that has the right methods, described by two traits in the runtime (runtime/iterator.pep, documented in the Iterators reference):

export trait Iterator<T> {
    next(): bool;    # advance; true if there is a current element
    current(): T;    # the element next() just moved to
}

export trait Iterable<T> {
    iterate(): Iterator<T>;
}
  • An iterator steps through elements. next() moves to the next element and returns true, or returns false when there are no more (and keeps returning false after that). current() returns the element next() moved to.
  • An iterable is a collection that hands out a fresh iterator from iterate(). Array<T> returns an ArrayIterator<T>, String returns a StringIterator, and HashMap<K, V> and HashSet<K> return a HashMapIterator<K, V> and a HashSetIterator<K>.

for (var x in source) { body } behaves exactly like this:

{
    var it = source.iterate();   # or `var it = source;` when source is itself an iterator
    while (it.next()) {
        var x = it.current();
        { body }
    }
}

The loop chooses by the methods the source’s type has, like a C++ range-based for: it uses iterate() if the type has one, and otherwise uses the source itself as the iterator if it has both next() and current(). A type with neither is a compile error:

main.pep:4:19: error: type 'Foo' is not iterable: implement Iterable<T> (iterate()) or Iterator<T> (next() and current())
    4 |     for (var x in f) {
      |                   ^

HashMap<K, V> yields each entry as a tuple (K, V), so for (var (name, age) in ages) names both parts; HashSet<K> yields its members. Both visit entries in no particular order. Changing a value (ages[name] = … for a key already present) is fine while looping, but adding or removing an entry stops the program at the iterator’s next step.

Adapters

Iterator<T> has default methods that wrap an iterator in another one. Each works lazily, one element at a time as the loop asks for it:

  • map(transform) yields transform(x) for each element x; its element type is whatever transform returns.
  • filter(keep) yields only the elements for which keep(x) is true.
  • enumerate() yields (index, x) tuples, counting from 0 in ulong; enumerate<int>() counts in int instead.
  • zip(other) yields (x, y) pairs, one element from each iterator in step, and stops when either runs out.

They return iterators, so they chain, and a for loop can use the result directly; destructure a tuple to name its parts. to_array() runs an iterator to the end and collects what is left into a new Array:

struct Person {
    String name;
    int age;
    construct(this.name = "", this.age = 0) {}
}

main(): int {
    var people = Array<Person>();
    people.append(Person("Ada", 36));
    people.append(Person("Bo", 12));
    people.append(Person("Cyd", 40));

    # filter keeps the adults, map turns each one into a name.
    var names = Array<String>();
    for (var name in people.iterate().filter((p) => p.age >= 18).map((p) => p.name)) {
        names.append(name);
    }
    assert(names.count == 2, "two adults");
    assert(names[0] == "Ada", "Ada first");
    assert(names[1] == "Cyd", "then Cyd");

    # Adapters work on any Iterator, including a String's.
    var kept = 0;
    for (var c in "hello".iterate().filter((c) => c != "l")) {
        kept += 1;
    }
    assert(kept == 3, "h, e, o");

    # enumerate pairs each element with its position, and zip pairs two
    # iterators in step; destructure the tuple to name both parts.
    var ranked = "";
    for (var (i, person) in people.iterate().enumerate()) {
        ranked += $"{i}:{person.name} ";
    }
    assert(ranked == "0:Ada 1:Bo 2:Cyd ", "numbered from 0");
    var seats = ["window", "aisle"];
    var seated = 0;
    for (var (person, seat) in people.iterate().zip(seats.iterate())) {
        seated += 1;   # Cyd gets no seat: zip stops at the shorter
    }
    assert(seated == 2, "two pairs");

    # to_array collects what an iterator yields.
    var ages = people.iterate().map((p) => p.age).to_array();
    assert(ages.count == 3 && ages[2] == 40, "every age");
    return 0;
}

Every type that declares Iterator<T> gets them: ArrayIterator<T> (from Array.iterate()), StringIterator, ranges, and your own iterators once you write extend YourType with Iterator<T>;. The adapter holds its own copy of the iterator it wraps (the runtime’s iterators are value structs, boxed when they become an Iterator<T>), so a chain built from a temporary stays alive while the loop runs, and it.map(f) and it.to_array() leave it where it was. They are generic methods (see Generics), so they are not callable through an Iterator<T> trait value.

Reading files and directories

Input that can fail partway, such as a directory listing or a file read line by line, yields a Result per element: an IoResult<T> from the io module. Handle each error where it happens; there is nothing to check after the loop.

import [File, Directory, Path] from io;

main(): int {
    var dir = Directory.temp().join("pepper-iteration-streams-example");
    Directory.remove_all(dir);
    assert(Directory.create_all(dir).is_ok(), "created the folder");
    assert(File.write_text(dir.join("notes.txt"), "first\nsecond\r\nthird").is_ok(), "wrote notes");
    assert(File.write_text(dir.join("todo.txt"), "").is_ok(), "wrote todo");

    # A Directory yields its entries one at a time, each an IoResult<DirEntry>.
    var names = 0;
    for (var entry in Directory.open(dir).value()) {
        if (entry.is_err()) {
            print(entry.error().message);
            break;
        }
        assert(entry.value().is_file, "both entries are files");
        names += 1;
    }
    assert(names == 2, "two entries");

    # A File's lines() yields each line, without its line ending.
    var file = File.open(dir.join("notes.txt")).value();
    var lines = Array<String>();
    for (var line in file.lines()) {
        lines.append(line.value());
    }
    assert(lines.count == 3 && lines[1] == "second", "three lines, \\r\\n removed");

    # Adapters work on streams too: number the lines.
    var again = File.open(dir.join("notes.txt")).value();
    var last = "";
    for (var (i, line) in again.lines().enumerate()) {
        last = $"{i}: {line.value()}";
    }
    assert(last == "2: third", "the last line, numbered");

    assert(Directory.remove_all(dir).is_ok(), "cleaned up");
    return 0;
}
  • A Directory is iterable. for (var entry in dir) reads the listing one entry at a time (Directory.iterate() returns a DirectoryIterator), without building an array the way dir.entries() does. Each element is an IoResult<DirEntry>. A directory that cannot be read yields one error and then ends.
  • lines() yields each line of an open File, or of any other io.Reader such as a socket, from what it reads next, as an IoResult<String> without the \n or \r\n. The last line counts even without a line ending. A read error, or a line that is not valid UTF-8, is yielded as an error and ends the lines.
  • chunks(size) yields the rest of a Reader in arrays of at most size bytes, one read each, as IoResult<u8[]>s.
  • An error ends the sequence: after an Err element, next() returns false.
  • listener.incoming() accepts connections on a TcpListener, yielding each one as an IoResult<TcpStream>. It is the exception to the rule above: a failed accept is yielded and the loop goes on, since the next connection may succeed. It ends only once the listener is closed, so a server loop breaks or returns when it is done.
  • Leaving early cleans up. The listing is closed when the loop ends, including by break or return, because the iterator is released then.

Writing your own iterator

To make your own type work in a for loop, give it next() and current() (an iterator), or give it an iterate() method that returns such a type (an iterable):

# An iterator: next() advances, current() returns the element.
struct Countdown {
    int remaining;
    int now;

    construct(this.remaining = 0) {}

    bool next() {
        if (remaining == 0) {
            return false;
        }
        now = remaining;
        remaining = remaining - 1;
        return true;
    }

    int current() {
        return now;
    }
}

# Declaring the trait documents the intent; the loop goes by the methods.
extend Countdown with Iterator<int>;

# An iterable: iterate() hands out a fresh iterator for each loop.
struct Evens {
    int limit;

    construct(this.limit = 0) {}

    EvensIterator iterate() {
        var it = EvensIterator();
        it.limit = limit;
        it.now = -2;
        return it;
    }
}

struct EvensIterator {
    int limit;
    int now;

    bool next() {
        now = now + 2;
        return now < limit;
    }

    int current() {
        return now;
    }
}

extend EvensIterator with Iterator<int>;
extend Evens with Iterable<int>;

# A generic function can loop over any type that supports the protocol.
sum<C>(C values): int {
    var total = 0;
    for (var v in values) {
        total = total + v;
    }
    return total;
}

main(): int {
    var total = 0;
    for (var n in Countdown(3)) {
        total = total + n;
    }
    assert(total == 6, "3 + 2 + 1");

    # A loop over an iterator in a variable uses that object up.
    var c = Countdown(2);
    for (var n in c) {
        total = total + n;
    }
    assert(!c.next(), "the countdown is exhausted");

    # An iterable starts a new pass every time.
    var evens = Evens(10);
    assert(sum<Evens>(evens) == 20, "0 + 2 + 4 + 6 + 8");
    assert(sum<Evens>(evens) == 20, "a second pass sees the same elements");

    var nums = Array<int>();
    nums.append(30);
    nums.append(12);
    assert(sum<Array<int>>(nums) == 42, "arrays work too");
    return 0;
}

Some things to keep in mind:

  • Declaring the trait is optional. The loop looks only at which methods a type has. Writing extend Countdown with Iterator<int>; documents your intent, and the compiler then checks that Countdown has both next() and current() (see Traits and extensions). Declaring it also lets the type be used as an Iterator<int> value, and a for loop over such a value calls the methods through the trait. It also gives the type the adapters (map, filter, enumerate, zip) and to_array().
  • Looping over an iterator uses it up. When the source is itself an iterator stored in a variable, the loop drives that variable in place, so it is exhausted afterwards, and a loop left with break can be resumed by looping over it again. A const variable holding a value iterator cannot be looped over, since next() changes it. An iterable starts a new, independent pass every time you loop over it.
  • Generic code can loop over a type parameter; each specialization checks the protocol for its own type (see Generics).
  • Calling current() out of turn panics in the runtime’s iterators: before the first next() you get ArrayIterator.current() before next(), and after next() returned false you get ArrayIterator.current() past the end. Your own iterators should follow the same rule.

Modifying a collection while you loop over it

Changing an Array or a String while a loop is iterating over it stops the program with a panic, because the iterator could otherwise skip or repeat elements, or read past the end:

var nums = Array<int>();
nums.append(1);
nums.append(2);
for (var x in nums) {
    nums.append(x);   # panics: Array modified during iteration
}

Every method that changes an Array counts, including storing an element with nums[i] = v. For a String, append and grow count, and the message is String modified during iteration. The check happens on the iterator’s next step, so modifying the collection after the loop is finished is fine.

To change elements as you go, loop over an index with while and store through it, or collect the changes in a second array and apply them after the loop.

Loop scope

The loop variable and anything declared in the loop body exist only inside the loop. Each pass gets a new loop variable, and struct values declared in the body are released at the end of every pass, whether the pass ends normally or by continue, break, or return. Leaving the loop early also releases the hidden iterator, so a destructor on a custom iterator runs at that point.

Using the loop variable or a body local after the loop is an error, the same as for any name that is out of scope:

main.pep:6:12: error: use of undeclared identifier 'i'
    6 |     return i;
      |            ^

The loop variable, or a local in the loop body, may have the same name as a variable declared before the loop. It hides the outer variable inside the loop, and the outer variable keeps its value afterwards.