Runes and UTF-8

A rune is an int32 holding one Unicode code point; ranging a string decodes UTF-8, while indexing it returns a single byte.

Syntaxfor i, r := range s { } — i is a byte position, r is a rune (int32)

Unicode gives every character in every script a number, called a code point. The letter h is 104. The letter é is 233. Go's name for one of those numbers is rune. The language specification states that rune is an alias for int32, so a rune really is just a 32-bit whole number, and byte is likewise an alias for uint8. You write one with single quotes: 'é'. Double quotes make a string, single quotes make a rune, and the difference matters.

UTF-8 is how runes become bytes

A string stores bytes, not runes. UTF-8 is the rule for turning a code point into between one and four bytes. Code points below 128 take one byte, which is why plain English text has one byte per character. é takes two. Many emoji take four.

range decodes, indexing does not

A for loop with range over a string walks it one character at a time, decoding UTF-8 as it goes. It hands you two values: the byte position the character starts at, and the rune itself. Those positions are not 0, 1, 2 in a row; they jump by the width of each character. Indexing with s[1] skips all of that and returns the raw byte at position 1, which may be the middle of a character. utf8.RuneCountInString(s) counts characters, where len(s) counts bytes.

What trips people up

Slicing a string slices bytes, so s[:3] can split a character in two. Convert with []rune(s) when you need to index by character; that copies the text into one int32 per character.

Example

Example · go
package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	s := "héllo"

	// len counts bytes. RuneCountInString counts characters.
	fmt.Println(len(s), utf8.RuneCountInString(s))

	// A rune literal uses single quotes. A rune is a number: int32.
	r := 'é'
	fmt.Printf("%T %d %U\n", r, r, r)

	// range decodes UTF-8. i is a byte position, c is the rune there.
	for i, c := range s {
		fmt.Printf("%d %c %d\n", i, c, c)
	}

	// Indexing does not decode. s[1] is the first byte of the two-byte é.
	fmt.Println(s[1])
}

// Output:
// 6 5
// int32 233 U+00E9
// 0 h 104
// 1 é 233
// 3 l 108
// 4 l 108
// 5 o 111
// 195

When to use it

  • Count the characters a user typed, so a 20-character limit is not 20 bytes.
  • Truncate a name for display without leaving half a character at the end.
  • Reject text from an upload that is not valid UTF-8 before storing it.
  • Check that every character of a username is a letter or a digit, in any script.

More examples

Indexing by character needs []rune

String slicing counts bytes, so taking the first three bytes of naive-with-a-diaeresis lands inside the i-with-two-dots and produces broken text. []rune(s) copies the string into one int32 per character, and then position 2 really is the third character. Converting a single rune back with string(r) gives its UTF-8 bytes. Reach for []rune when you truncate text for display or reverse it.

Example · go
package main

import "fmt"

func main() {
	s := "naïve"

	// Slicing a string slices bytes. This cuts the ï in half.
	fmt.Printf("%q\n", s[:3])

	// []rune(s) copies the text into one int32 per character.
	rs := []rune(s)
	fmt.Printf("%q\n", string(rs[:3]))

	// string(r) turns one rune back into its UTF-8 bytes.
	fmt.Println(len(string(rs[2])))
}

// Output:
// "na\xc3"
// "naï"
// 2

Invalid UTF-8 does not stop a range loop

Nothing stops a Go string holding bytes that are not valid UTF-8, because a string is just bytes. The specification says a range loop over such a string yields 0xFFFD, the replacement character, and moves on by one byte. So a loop never fails, it quietly substitutes. Check with utf8.ValidString before you trust text that arrived from a file, a form or a network.

Example · go
package main

import (
	"fmt"
	"unicode/utf8"
)

func main() {
	// A string can hold bytes that are not valid UTF-8.
	bad := "a\xffb"
	fmt.Println(utf8.ValidString(bad))

	// range does not fail. It yields U+FFFD and steps on one byte.
	for i, c := range bad {
		fmt.Printf("%d %U\n", i, c)
	}

	// RuneError is that replacement character, so you can test for it.
	fmt.Printf("%U\n", utf8.RuneError)
}

// Output:
// false
// 0 U+0061
// 1 U+FFFD
// 2 U+0062
// U+FFFD

Asking questions about one character

The unicode package takes a rune and answers one question about it, across every script rather than just A to Z. IsLetter, IsDigit and ToUpper all work on the Scandinavian o-with-a-stroke the same way they work on ASCII. This is how you validate a username or fold a search term, instead of comparing against 'a' and 'z' by hand.

Example · go
package main

import (
	"fmt"
	"unicode"
)

func main() {
	// unicode asks about one rune at a time, in any script.
	for _, c := range "A1ø" {
		fmt.Printf("%c letter=%v digit=%v upper=%c\n", c, unicode.IsLetter(c), unicode.IsDigit(c), unicode.ToUpper(c))
	}
}

// Output:
// A letter=true digit=false upper=A
// 1 letter=false digit=true upper=1
// ø letter=true digit=false upper=Ø

Related

go stringsgo rangego booleans conversiongo slicesgo for

Discussion

  • Be the first to comment on this lesson.