Generators & yield
Produce values lazily, one at a time, so you can process streams bigger than memory.
A generator is a function that yields values instead of returning once. Each yield hands a value to the caller and pauses the function, keeping all its local state. On the next request it resumes right where it left off.
def count_up(n):
i = 0
while i < n:
yield i # pause here, hand out i
i += 1
for x in count_up(3):
print(x) # 0, 1, 2Why this is powerful
Generators are lazy: they compute each value only when asked and never hold the whole sequence in memory. That means you can iterate over a ten-gigabyte file, an infinite sequence, or a live stream while using almost no RAM.
Generator expressions
The comprehension's leaner cousin uses parentheses: sum(x*x for x in range(1000)) streams the squares straight into sum() without building a list first.
yield from
yield from another_generator() delegates to a sub-generator, flattening nested producers cleanly.
Example
def read_records(lines):
"""Stream 'key=value' lines into dicts, lazily."""
for line in lines:
line = line.strip()
if not line or line.startswith('#'):
continue # skip blanks and comments
key, _, value = line.partition('=')
yield {key.strip(): value.strip()}
def take(gen, n):
"""yield the first n items from any generator."""
for i, item in enumerate(gen):
if i >= n:
return
yield item
raw = [
'# config',
'host = localhost',
'',
'port = 8080',
'debug = true',
]
# Nothing is computed until we iterate — and only what we ask for
records = read_records(raw)
for rec in take(records, 2):
print(rec)
# Generator expression feeding a reducer — no intermediate list
total = sum(len(line) for line in raw if line and not line.startswith('#'))
print('total chars of real lines:', total)
# Infinite generator, safely bounded by take()
def naturals():
n = 1
while True:
yield n
n += 1
print('first 5 naturals:', list(take(naturals(), 5)))
# Output:
# {'host': 'localhost'}
# {'port': '8080'}
# total chars of real lines: 36
# first 5 naturals: [1, 2, 3, 4, 5]When to use it
- A large CSV reader uses a generator to yield one parsed row at a time, keeping memory flat.
- An infinite ID generator yields successive unique integers for new database rows on demand.
- A chained data pipeline uses generator expressions to filter and transform records without materialising intermediate lists.
More examples
Generator with send() for coroutine
Uses send() to push values into a generator, enabling a stateful running total without a class.
def accumulator():
total = 0
while True:
value = yield total
if value is None:
break
total += value
acc = accumulator()
next(acc) # prime the generator
print(acc.send(10)) # 10
print(acc.send(20)) # 30
print(acc.send(5)) # 35yield from to delegate
Uses 'yield from' to delegate to a recursive generator call, flattening an arbitrarily nested list.
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item)
else:
yield item
data = [1, [2, [3, 4]], 5, [6]]
print(list(flatten(data))) # [1, 2, 3, 4, 5, 6]Chained generator pipeline
Composes three generators into a lazy pipeline; no intermediate list is ever created.
def read_ints(n):
yield from range(n)
def only_even(seq):
return (x for x in seq if x % 2 == 0)
def squared(seq):
return (x ** 2 for x in seq)
pipeline = squared(only_even(read_ints(10)))
print(list(pipeline)) # [0, 4, 16, 36, 64]
Discussion