io.Reader and io.Writer

Two one-method interfaces in the io package let files, network connections, strings and buffers all be used interchangeably.

The io package of the standard library declares two one-method interfaces. Nearly everything in Go that moves bytes uses them, which is why a function written for one source of bytes works with all of them.

type Reader interface {
	Read(p []byte) (n int, err error)
}

type Writer interface {
	Write(p []byte) (n int, err error)
}

A byte is one unit of data, and []byte is a slice of them. error is a built-in interface type; a nil error means nothing went wrong. Files, network connections, HTTP request bodies, os.Stdout, strings.Reader and strings.Builder all satisfy one or both of these.

The contract

Read is given a slice to fill. It copies in up to len(p) bytes and returns how many it copied. It may return fewer than asked for. When there is no more data it returns the error io.EOF, which stands for end of file and is not a failure. Write must write all of p or return an error saying why it could not.

Because both are interfaces, wrappers compose: io.Copy(dst, src) moves everything from a reader to a writer, and io.ReadAll(r) returns all of it as a slice. io.ReadAll was added in Go 1.16; the older ioutil.ReadAll is deprecated.

Longhand and shorthand

These two lines do the same thing. The first formats text and calls Write for you.

fmt.Fprintf(w, "id=%d\n", 7)
w.Write([]byte(fmt.Sprintf("id=%d\n", 7)))

Example

Example · go
package main

import (
	"fmt"
	"io"
	"os"
	"strings"
)

// upperWriter satisfies io.Writer, whose only method is
//	Write(p []byte) (n int, err error)
type upperWriter struct {
	dst io.Writer
}

func (u upperWriter) Write(p []byte) (int, error) {
	// Upper-casing ASCII keeps the length the same, so the count
	// returned is a valid count of bytes from p.
	return u.dst.Write([]byte(strings.ToUpper(string(p))))
}

func main() {
	// strings.NewReader returns a value with a Read method, so it
	// is an io.Reader.
	src := strings.NewReader("hello pipes\n")

	// io.Copy pulls from the reader and pushes into the writer.
	n, err := io.Copy(upperWriter{dst: os.Stdout}, src)
	if err != nil {
		fmt.Fprintln(os.Stderr, "copy failed:", err)
		return
	}
	fmt.Println("bytes copied:", n)
}

// Output:
// HELLO PIPES
// bytes copied: 12

When to use it

  • Write one upload handler that works with a file, a network body or a test string.
  • Count or hash bytes as they stream past by wrapping an io.Writer.
  • Copy a response straight to disk with io.Copy instead of buffering it all in memory.

More examples

Reading by hand, one buffer at a time

This is the loop every higher-level helper is built on: ask for a buffer's worth, use what you got, stop at io.EOF. Write it by hand when you must process data in fixed-size pieces instead of loading all of it into memory.

Example · go
package main

import (
	"fmt"
	"io"
	"strings"
)

func main() {
	r := strings.NewReader("hello")
	buf := make([]byte, 3) // Read never gives more than this

	for {
		n, err := r.Read(buf)

		// Use the bytes first: n can be above zero even when err
		// is set.
		if n > 0 {
			fmt.Printf("read %d: %q\n", n, buf[:n])
		}
		if err == io.EOF {
			fmt.Println("end of input")
			break
		}
		if err != nil {
			fmt.Println("read failed:", err)
			break
		}
	}
}

// Output:
// read 3: "hel"
// read 2: "lo"
// end of input

Writing to two places at once

Writing your own io.Writer takes one method, and io.MultiWriter fans a single stream out to several of them. Use this to count or hash bytes while they pass through, or to send the same log output to a file and the terminal.

Example · go
package main

import (
	"fmt"
	"io"
	"os"
	"strings"
)

// countWriter counts bytes and discards them. Write returns the
// number of bytes it accepted.
type countWriter struct {
	n int
}

func (c *countWriter) Write(p []byte) (int, error) {
	c.n += len(p)
	return len(p), nil
}

func main() {
	var c countWriter
	var b strings.Builder

	// io.MultiWriter sends every write to both destinations.
	w := io.MultiWriter(&c, &b)

	// Shorthand: Fprintf formats, then calls w.Write.
	fmt.Fprintf(w, "id=%d\n", 7)

	// Longhand: the same write done by hand.
	if _, err := w.Write([]byte("done\n")); err != nil {
		fmt.Fprintln(os.Stderr, "write failed:", err)
		return
	}

	fmt.Printf("%d bytes, buffer holds %q\n", c.n, b.String())
}

// Output:
// 10 bytes, buffer holds "id=7\ndone\n"

One function, any source of bytes

Taking io.Reader instead of a filename or a string is what makes a function testable: production code passes a file or a network body, and a test passes strings.NewReader. Be aware that io.ReadAll holds the whole input in memory, so cap or stream anything that could be large.

Example · go
package main

import (
	"fmt"
	"io"
	"strings"
)

// describe takes io.Reader, so the same function serves a string,
// a file on disk, or an HTTP request body.
func describe(r io.Reader) (string, error) {
	// io.ReadAll reads until io.EOF and returns every byte. It was
	// added in Go 1.16; ioutil.ReadAll is the deprecated form.
	b, err := io.ReadAll(r)
	if err != nil {
		return "", err
	}
	return fmt.Sprintf("%d bytes: %s", len(b), b), nil
}

func main() {
	out, err := describe(strings.NewReader("ready"))
	if err != nil {
		fmt.Println("read failed:", err)
		return
	}
	fmt.Println(out)
}

// Output:
// 5 bytes: ready

Related

go small interfacesgo interface basicsgo slicesgo writing jsongo file server

Discussion

  • Be the first to comment on this lesson.