Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

HTTP

The http module has an HTTP/1.1 and HTTP/2 client and server, over plain TCP or TLS. Both run in tasks: a request that waits parks only its task. Over TLS the two sides pick HTTP/2 through ALPN when both have it (see HTTP/2); the Request and Response types, and the code that uses them, are the same for either version.

HTTP needs tasks and sockets, so it is not available on WebAssembly (--target wasm32-wasip1, see WebAssembly).

import async;
import http;
import io;

# Answers every request: `/hello` greets, anything else is not found.
handle(request: http.Request): http.Response {
    if (request.path() == "/hello") {
        var response = http.Response(200).set_text("hello, " + request.query());
        response.headers["Content-Type"] = "text/plain";
        return response;
    }
    return http.Response(404).set_text("nothing at " + request.path());
}

main(): int {
    async.run(
        () => {
            # Port 0 asks the system for any free port.
            var server = http.Server.bind(io.SocketAddress.parse("127.0.0.1:0").value()).value();
            var port = server.local_address().value().port();
            # serve() runs until async.run ends, so it gets a task of its own.
            async.spawn<void>(() => {
                server.serve(handle);
            });

            var base = "http://127.0.0.1:" + to_string(port);
            var client = http.Client();
            var response = client.get(base + "/hello?pepper").value();
            assert(response.status == 200, "found");
            assert(response.text() == "hello, pepper", "the handler's text");
            assert(response.headers["content-type"] == "text/plain", "names ignore case");

            var missing = client.get(base + "/elsewhere").value();
            assert(missing.status == 404 && missing.reason == "Not Found", "not found");

            var posted = client.post(base + "/hello", "text/plain", "a body").value();
            assert(posted.is_success(), "any method reaches the handler");
        }
    );
    return 0;
}

Messages

  • A Request has a method, a url, headers and a body. For a client the url is absolute (https://example.com/items?page=2); a server handler gets the request target (/items?page=2). uri() parses either as a Uri, and path() and query() return its parts.
  • A Response has a status, a reason, headers and a body. is_success() tests for a 2xx status.
  • Headers maps each field name to its values. Names ignore case (they are stored lowercased). headers["Accept"] reads the first value ("" when there is none) and headers["Accept"] = "text/html" replaces the field; get(name) returns an optional, add keeps earlier values, values(name) returns them all, and names() lists the fields in no defined order.
  • set_text and set_body fill a request or response and return it, so they chain. body() returns the bytes, for any body including binary ones; text() decodes them as UTF-8, replacing invalid bytes with U+FFFD.
  • set_reader(source) gives a request or response a body streamed from any io.Reader (a file, a compress.CompressReader) instead of one held in memory; see Streaming bodies.

URLs

Uri (in the prelude, so no import is needed) parses any URI reference following RFC 3986: an absolute URI with any scheme (https:, mailto:, file:) or a relative reference such as a request target. It splits the text into scheme, user_info, host, port, path, query and fragment; the parts that can be missing are optionals. Components keep their percent-escapes, and Uri.decode turns one into text; Uri.encode escapes text for a path segment or query value, and Uri.encode_form and Uri.decode_form write and read form data (see Encodings and UUIDs). base.resolve(reference) resolves a relative reference against a base URI, as a browser resolves a link (RFC 3986 section 5.2).

main(): int {
    # Any scheme parses; the scheme is lowercased and nothing else changes.
    var uri = Uri.parse("HTTPS://user@example.com:8443/caf%C3%A9?page=2#top").value();
    assert(uri.scheme == "https" && uri.host.value() == "example.com", "scheme and host");
    assert(uri.port.value() == 8443 && uri.user_info.value() == "user", "port and user information");
    assert(uri.path == "/caf%C3%A9" && uri.query.value() == "page=2", "path and query keep their escapes");
    assert(Uri.decode(uri.path).value() == "/café", "decoded path");
    assert(uri.path_and_query() == "/caf%C3%A9?page=2", "what an HTTP request line names");

    # Missing parts are Empty: a mailto URI has no authority.
    var mail = Uri.parse("mailto:someone@example.com").value();
    assert(mail.host.is_empty() && mail.path == "someone@example.com", "no authority");

    # A relative reference has no scheme, like a request target on a server.
    var target = Uri.parse("/items?page=2").value();
    assert(!target.is_absolute() && target.path == "/items", "relative");

    # resolve() turns a relative reference, such as a redirect's Location,
    # into the URI it names.
    var page = Uri.parse("https://example.com/docs/guide/intro").value();
    assert(page.resolve(Uri.parse("../api?v=2").value()).to_string() == "https://example.com/docs/api?v=2", "resolved");

    assert(Uri.parse("http://example.com/a b").is_err(), "a space must be percent-encoded");
    return 0;
}

Client

  • Client() trusts the system’s root certificates for https:// URLs. Client(connector) uses a tls.TlsConnector of your own, for instance one that trusts a private certificate authority.
  • get(url), post(url, content_type, body) and send(request) open a connection, send the request and read the whole response, up to MAX_RESPONSE_BODY (64 MiB). open(request) returns at the header fields and leaves the body to read as a stream (see Streaming bodies). The client adds Host, Content-Length (or Transfer-Encoding: chunked) and Connection: close.
  • Any status is a successful result: check status. An error means there was no response: the URL is not http:// or https:// (is_invalid_input()), the connection or TLS handshake failed, or the response was not valid HTTP.
  • Redirects (301, 302, 303, 307, 308) are followed, up to max_redirects (10) per request, and response.url is where the response came from. A 303, and a 301 or 302 after a POST, becomes a GET without the body; a 307 or 308 repeats the method and body, except that a body from set_reader cannot be sent twice, so that redirect is returned. Leaving the origin drops Authorization, Proxy-Authorization and Cookie. Client(0) returns every redirect as it is.
  • Every request from a Client uses a new connection; a Session keeps them open (see Reusing connections).
  • For https:// the client offers HTTP/2 and HTTP/1.1 through ALPN, on its own connector or the one it was given, and speaks whichever the server picks.

Server

  • Server.bind(address) listens for HTTP; Server.bind_tls(address, acceptor) serves HTTPS with a tls.TlsAcceptor’s certificate.
  • serve(handler) accepts connections on every worker of async.run and serves each in a task of its own, calling handler for each request with the body read in full. It returns when async.run ends. The handler is a shared (Request) => Response, since every worker calls it.
  • Connections stay open between requests unless the client asks to close or speaks HTTP/1.0. Chunked request bodies and Expect: 100-continue are understood.
  • The server adds Content-Length to each response (or, for a body from set_reader of unknown length, Transfer-Encoding: chunked), and the standard reason phrase when reason is empty (reason_phrase(status) returns it).
  • A request it cannot parse gets 400 Bad Request, header fields over 64 KiB get 431, and a body over MAX_REQUEST_BODY (16 MiB) gets 413.
  • Server.bind_tls offers HTTP/2 and HTTP/1.1 through ALPN, and a plain-TCP client that opens with the HTTP/2 preface (prior knowledge, or h2c) is served HTTP/2 too. See HTTP/2.

Streaming bodies

A body need not fit in memory. Both directions stream through io.Reader, the trait of files, sockets and the compress readers:

  • Responses to a client: client.open(request) returns the Response as soon as its header fields arrive. response.reader() is then a BodyReader that reads the body from the connection as it arrives, with no size limit, undoing chunked framing, and ends where the body ends. Read it with read, chunks or lines, or wrap it in a compress.DecompressReader. The connection closes at the end of the body, at close(), or when the last reference to the reader goes.
  • Requests from a client: request.set_reader(source) sends the body chunked; set_reader(source, length) sends it with Content-Length.
  • Responses from a server: a handler returns Response(200).set_reader(source), sent chunked to an HTTP/1.1 client (or with Content-Length when a length is given). An HTTP/1.0 client gets the body unframed, and the connection closes after it.
import async;
import http;
import io;

# `/download` streams a body of unknown length, which goes out chunked;
# `/upload` reports how many bytes it received.
handle(request: http.Request): http.Response {
    if (request.path() == "/download") {
        var data = "pepper\n".repeat(100000).bytes();
        return http.Response(200).set_reader(io.ByteReader(data));
    }
    return http.Response(200).set_text(to_string(request.body().count));
}

main(): int {
    async.run(
        () => {
            var server = http.Server.bind(io.SocketAddress.parse("127.0.0.1:0").value()).value();
            var port = server.local_address().value().port();
            async.spawn<void>(() => {
                server.serve(handle);
            });
            var base = "http://127.0.0.1:" + to_string(port);
            var client = http.Client();

            # open() returns at the header fields; the body is read as it
            # arrives and written to a file a chunk at a time.
            var response = client.open(http.Request("GET", base + "/download")).value();
            assert(response.headers["transfer-encoding"] == "chunked", "no length up front");
            var path = io.Directory.temp().join("pepper-download.txt");
            var file = io.File.create(path).value();
            for (var chunk in response.reader().chunks(65536)) {
                file.write_all(chunk.value());
            }
            file.close();

            # Send the file back as a request body read from the file.
            var upload = http.Request("PUT", base + "/upload").set_reader(io.File.open(path).value());
            assert(client.send(upload).value().text() == "700000", "every byte arrived");
            io.File.remove(path);
        }
    );
    return 0;
}

Reusing connections

client.session() returns a Session, which sends requests with the client’s settings and keeps each connection open after its response (HTTP/1.1 keep-alive). The next request to the same scheme, host and port uses it again, saving the TCP and TLS handshakes:

import async;
import http;
import io;

main(): int {
    async.run(
        () => {
            var server = http.Server.bind(io.SocketAddress.parse("127.0.0.1:0").value()).value();
            var port = server.local_address().value().port();
            async.spawn<void>(() => {
                server.serve((request) => {
                    return http.Response(200).set_text("page " + request.query());
                });
            });
            var base = "http://127.0.0.1:" + to_string(port);

            var session = http.Client().session();
            for (var page in 1..>3) {
                var response = session.get(base + "/items?" + to_string(page)).value();
                assert(response.text() == "page " + to_string(page), "each page");
            }
            # One connection carried all three requests, and waits for more.
            assert(session.idle_count == 1, "kept open");
            session.close();
        }
    );
    return 0;
}
  • A connection goes back to the session when its response body has been read to its end (send reads it; for open, read the BodyReader to its end), unless the server said Connection: close.
  • max_idle_per_host (4) connections wait for each scheme, host and port; one idle longer than idle_timeout (90 seconds) is closed instead of used. idle_count says how many wait, and close() closes them.
  • A request that finds its reused connection closed by the server goes again on a new one, unless its body came from set_reader.
  • An HTTP/2 connection is not idle and taken in turn: every request to its origin shares it at once, each on a stream of its own. idle_count counts it while no request is using it.
  • A Session is not [Shared]: use one per thread or task. A Client is shared, and connections, which belong to one thread, cannot be. Tasks started with spawn_local may share a session, and over HTTP/2 their requests go out on one connection at the same time.

HTTP/2

HTTP/2 (RFC 9113) carries many requests on one connection at once, as streams, with header fields compressed (HPACK, RFC 7541) and each body flow-controlled so that one large transfer does not hold up the rest.

  • Choosing it: over TLS, the client offers h2 and http/1.1 through ALPN and Server.bind_tls accepts both, preferring h2, so two Pepper programs speak HTTP/2 with no setting to change. A server or client that offers only HTTP/1.1 gets HTTP/1.1. On plain TCP the client speaks HTTP/1.1; the server also takes HTTP/2 from a client that starts with its preface (prior knowledge).
  • Client: a Session keeps one HTTP/2 connection per origin, and requests from several tasks share it at once. A request outside a session opens a connection that closes after its body. A connection the server ends with GOAWAY is replaced by a new one for the next request.
  • Server: each stream is a request, answered by the same handler in a task of its own, so the requests of one connection run at the same time. The handler sees the Host field from :authority. A server lets a client open 100 streams at once.
  • Messages: field names go lowercase on the wire, as HTTP/2 requires, and fields that only mean something to HTTP/1.1 (Connection, Transfer-Encoding, Upgrade, Keep-Alive) are left out. HTTP/2 has no reason phrase, so a response’s reason is the standard one for its status. Bodies stream through io.Reader exactly as for HTTP/1.1; closing a response’s reader early resets only its stream.

Current status

  • A server handler gets the request body read into memory in full, up to MAX_REQUEST_BODY (16 MiB).
  • HTTP/2 server push, stream priorities and the Upgrade: h2c handshake are not supported; a client speaks HTTP/2 only over TLS.

The reference pages for messages, the client, the server and Uri document every method.