Python · Lesson 8
Iterators and Generators
The iterator protocol behind every for loop, one-shot iterators, writing __iter__ and __next__, generator functions and yield, lazy and infinite sequences, yield from, generator expressions and pipelines, send and close, and the itertools functions that replace most hand-written loops.
In short
iter()andnext(); the end isStopIteration- Iterators are one-shot; lists are re-iterable
yieldmakes a lazy, pausable functionyield fromdelegates; genexps build pipelinesitertools:islice,pairwise,batched,groupby(sort first)
for loops, list(), sum(), max(), sorted(), in, unpacking (a, b = x), dict(zip(...)) and
", ".join(...) all work on lists, strings, dicts, files and ranges. None of them is special-cased for each
type. They all speak one small protocol, and any object that speaks it works with all of them. Generators
are the easy way to speak it, and itertools is the standard toolbox built on it.
If you know JavaScript, this is Symbol.iterator and function*. If you know C#, it's IEnumerable<T>,
yield return and LINQ's deferred execution. The ideas carry over directly; the differences are in the
details, and the details are what interviews ask about.
Every python snippet has a ▶ Try button. Predict the output first, then run it.
The protocol: iter and next
Two roles, two methods:
- An iterable is anything you can loop over. It has
__iter__(), which returns an iterator. The built-initer(x)calls it. - An iterator produces the values. It has
__next__(), which returns the next value or raisesStopIterationwhen there are no more. The built-innext(it)calls it.
nums = [10, 20]
it = iter(nums)
print(next(it))
print(next(it))
try:
next(it)
except StopIteration:
print("StopIteration: no more values")
print(next(it, "done")) # a default instead of the exception
A for loop is that protocol with the exception handled for you. for x in nums: print(x) does this:
nums = [10, 20]
it = iter(nums)
while True:
try:
x = next(it)
except StopIteration:
break
print(x)
The same idea in the three languages:
| Python | JavaScript | C# | |
|---|---|---|---|
| Get an iterator | iter(x) calls x.__iter__() | x[Symbol.iterator]() | x.GetEnumerator() |
| Next value | next(it) calls it.__next__() | it.next() returns { value, done } | it.MoveNext(), then it.Current |
| End of sequence | raises StopIteration | done: true | MoveNext() returns false |
| Generator | a def containing yield | function* with yield | a method with yield return |
| Delegate to another | yield from | yield* | foreach + yield return |
Python signals the end with an exception. That sounds expensive, but for loops and built-ins handle it in C,
and you rarely see it unless you call next() yourself.
Iterables vs iterators: one-shot
A list is iterable but not an iterator. Each iter(list) gives a fresh iterator starting at the
beginning, so you can loop over a list as often as you like. An iterator's __iter__ returns itself, so it
can be consumed only once:
nums = [1, 2, 3]
print(iter(nums) is iter(nums)) # two separate iterators
it = iter(nums)
print(iter(it) is it) # an iterator is its own iterator
print(list(it))
print(list(it)) # already used up
This is the most common iterator bug in real code: a function that walks its argument twice works with a list and silently breaks with a generator.
def average(values):
total = sum(values)
count = len(list(values))
return total / count if count else None
print(average([3, 4, 5]))
print(average(v for v in [3, 4, 5])) # sum() used everything up
The first call prints 4.0. The second prints None: sum consumed the generator, so list(values) was
empty. If you need two passes, materialize once (values = list(values)) and say so in the type hint
(Sequence instead of Iterable).
Many built-ins return iterators, not lists: map, filter, zip, enumerate, reversed, and file
objects. They're lazy and one-shot. range is different: it's a lazy iterable that you can loop over
again, measure with len, and test with in without looping.
squares = map(lambda n: n * n, [1, 2, 3])
print(type(squares).__name__)
print(list(squares), list(squares))
r = range(0, 10, 2)
print(list(r), list(r), len(r), 6 in r)
in on an iterator consumes it up to the match, which surprises people:
it = iter([1, 2, 3, 4])
print(3 in it)
print(list(it)) # 1, 2 and 3 are gone
Writing an iterator class
To make your own type iterable, give it __iter__. The classic textbook version splits the iterable from the
iterator, so the iterable can be looped over many times:
class Countdown:
def __init__(self, start: int) -> None:
self.start = start
def __iter__(self) -> "CountdownIterator":
return CountdownIterator(self.start)
class CountdownIterator:
def __init__(self, current: int) -> None:
self.current = current
def __iter__(self) -> "CountdownIterator":
return self
def __next__(self) -> int:
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
c = Countdown(3)
print(list(c), list(c))
for n in c:
print(n, end=" ")
print()
In practice you rarely write the iterator class. Make __iter__ a generator (next section) and Python
builds the iterator for you:
class Countdown:
def __init__(self, start: int) -> None:
self.start = start
def __iter__(self):
n = self.start
while n > 0:
yield n
n -= 1
print(list(Countdown(3)), sum(Countdown(4)), max(Countdown(5)))
Write a real iterator class when the iterator needs more methods than next: a peek(), a
push_back(), a .position for error messages. The Peekable exercise is that case.
One more built-in trick: iter(callable, sentinel) calls the function until it returns the sentinel. It turns
"read until empty" loops into iterators:
import io
f = io.StringIO("abcdefgh")
for chunk in iter(lambda: f.read(3), ""):
print(chunk)
Generators: functions that pause
A function that contains yield is a generator function. Calling it doesn't run the body. It returns a
generator object, which is an iterator. Each next() runs the body until the next yield, hands out that
value, and pauses there with all local variables intact.
def steps():
print(" start")
yield "a"
print(" after a")
yield "b"
print(" after b")
return "the end"
gen = steps() # nothing printed: the body hasn't started
print("created")
print(next(gen))
print(next(gen))
try:
next(gen)
except StopIteration as e:
print("finished, return value:", e.value)
print(list(gen)) # finished generators stay finished
Things to notice:
- Lazy. Code runs only when someone asks for the next value.
returnends the sequence. Its value travels on theStopIteration(e.value), andfor,list()and friends ignore it. It's only useful withyield from(below).- One-shot. A generator object is an iterator, so it can be consumed once. Call the function again for a fresh one.
C# developers: this is yield return, and return in a generator is like yield break (plus an optional
value). JavaScript developers: function* without the asterisk. Python decides that a function is a
generator by the presence of yield anywhere in its body.
Laziness, step by step
Chained generators pass values along one at a time. Watch the order:
def numbers():
for n in [1, 2, 3]:
print("produce", n)
yield n
def doubled(values):
for v in values:
print(" double", v)
yield v * 2
for result in doubled(numbers()):
print(" consume", result)
Each value travels the whole chain before the next one is produced: produce 1, double 1, consume 2, then produce 2... With list comprehensions instead, all three "produce" lines would print first. That's what makes generators work on data bigger than memory, and on data that never ends.
Infinite generators
Because nothing runs ahead, a generator can describe an endless sequence. The consumer decides how much to
take, usually with itertools.islice or a break:
from itertools import islice
def ids(prefix: str):
n = 1
while True:
yield f"{prefix}-{n:04d}"
n += 1
print(list(islice(ids("INV"), 3)))
for invoice_id in ids("ORD"):
if invoice_id.endswith("3"):
break
print("stopped at", invoice_id)
Calling list() on an infinite generator never returns. In an interview, say how the consumer bounds it.
Memory
A list comprehension builds every element; a generator holds one at a time. That's the reason to choose one over the other for big inputs:
import sys
as_list = [n * n for n in range(100_000)]
as_gen = (n * n for n in range(100_000))
print(sys.getsizeof(as_list) > 100_000, sys.getsizeof(as_gen) < 1_000)
print(sum(as_gen) == sum(as_list))
The list takes hundreds of kilobytes (more on a 64-bit machine); the generator object is a couple of hundred bytes, however long the sequence.
Two traps
Validation in a generator runs late. The body, including any argument checks at the top, waits for the
first next():
def chunks(items, size):
if size < 1:
raise ValueError("size must be at least 1")
for i in range(0, len(items), size):
yield items[i:i + size]
gen = chunks([1, 2, 3], 0)
print("no error yet")
try:
next(gen)
except ValueError as e:
print("only now:", e)
The fix is a plain function that validates and then returns an inner generator. The Message batches exercise uses it.
StopIteration inside a generator becomes RuntimeError. A bare next() that hits the end inside a
generator would silently end the outer generator, hiding the bug. Since Python 3.7 (PEP 479) it raises
instead:
def first_of_each(groups):
for group in groups:
yield next(iter(group)) # an empty group raises StopIteration here
try:
print(list(first_of_each([[1, 2], [], [3]])))
except RuntimeError as e:
print("RuntimeError:", e)
Use next(iter(group), None) with a default, or a for loop with break.
yield from
yield from iterable yields every value of another iterable, as if you'd written
for x in iterable: yield x. It's how a generator delegates, and recursion becomes natural:
def flatten(items):
for item in items:
if isinstance(item, list):
yield from flatten(item)
else:
yield item
print(list(flatten([1, [2, [3, 4]], [], 5])))
Walking a tree is the same shape:
tree = {"name": "src", "children": [
{"name": "app.py", "children": []},
{"name": "lib", "children": [{"name": "util.py", "children": []}]},
]}
def walk(node, depth=0):
yield " " * depth + node["name"]
for child in node["children"]:
yield from walk(child, depth + 1)
print("\n".join(walk(tree)))
yield from does more than the loop: it passes send() and close() through to the inner generator, and it
evaluates to the inner generator's return value:
def inner():
yield 1
yield 2
return "inner done"
def outer():
result = yield from inner()
print("inner returned:", result)
yield 3
print(list(outer()))
This is JavaScript's yield*. C# has no equivalent; you write foreach (var x in Inner()) yield return x;.
Generator expressions
A generator expression is a list comprehension in parentheses: (expr for x in xs if cond). It's lazy.
When it's the only argument of a call, the extra parentheses can go:
orders = [120, 80, 300, 45]
print(sum(o for o in orders if o >= 100))
print(max(len(word) for word in ["iter", "generator", "yield"]))
print(", ".join(str(o) for o in orders))
Laziness matters with any and all, which stop at the first answer they can give:
def is_big(n):
print("checking", n)
return n > 2
print(any(is_big(n) for n in [1, 3, 5, 7]))
print(any([is_big(n) for n in [1, 3, 5, 7]]))
The generator version checks 1 and 3 and stops. The list version builds the whole list first, checking all
four, and only then asks any.
One subtle rule: the first for's iterable is evaluated immediately, when the generator expression is
created. Everything else (the expression, the conditions) runs later, as values are pulled:
nums = [1, 2, 3]
factor = 10
gen = (n * factor for n in nums)
nums = [7, 8, 9] # too late: the generator already holds the old list
factor = 100 # not too late: factor is read as each value is made
print(list(gen))
It prints [100, 200, 300]: the old list's values, times the new factor. The same rule explains why a
generator expression over a list you later mutate sees the mutation (it holds the list object, not a copy).
Pipelines
Generators compose into pipelines, like Unix pipes: each stage pulls from the previous one, and nothing is stored in between.
from collections import Counter
log = """\
INFO start
ERROR db timeout
INFO retry
ERROR db timeout
WARN slow query
ERROR disk full
"""
lines = (line.strip() for line in log.splitlines())
entries = (line.partition(" ") for line in lines if line)
errors = (message for level, _, message in entries if level == "ERROR")
print(Counter(errors).most_common(2))
None of the first three lines does any work. Counter(...) pulls from errors, which pulls from entries,
which pulls from lines. With a real file it's the same code with open(path) as the source (a file object
is an iterator of lines), and it runs in constant memory on a 10 GB log. Keep the whole pipeline inside the
with open(...) block, because the generators read from the file lazily.
This is C#'s LINQ (lines.Where(...).Select(...), deferred until enumerated) and JavaScript's iterator
helpers (Iterator.from(x).filter(...).map(...)). Python writes it with generator expressions and
functions instead of method chains.
send and close, briefly
A generator can also receive values. gen.send(value) resumes it and makes the paused yield expression
evaluate to value. You must start it with next() first, to run it to the first yield:
def running_average():
total = 0.0
count = 0
average = None
while True:
value = yield average
total += value
count += 1
average = total / count
avg = running_average()
next(avg) # run to the first yield
print(avg.send(10))
print(avg.send(20))
print(avg.send(60))
This was the basis of Python's early coroutines. Today async def covers that use, and send is rare in
application code. You should recognise it, not reach for it.
gen.close() raises GeneratorExit at the paused yield, so finally blocks (and with blocks) inside the
generator run. That's how a generator that holds a resource cleans up when the consumer stops early:
def numbered(lines):
print("open")
try:
for i, line in enumerate(lines, 1):
yield f"{i}: {line}"
finally:
print("cleanup")
gen = numbered(["a", "b", "c"])
print(next(gen))
gen.close()
print("after close")
Unlike JavaScript's for...of, a Python for loop doesn't close the iterator when you break. CPython
closes a generator when the last reference to it disappears, which is usually right away, but not when you
keep the generator in a variable:
def source():
try:
yield 1
yield 2
finally:
print("source closed")
gen = source()
for x in gen:
print("got", x)
break
print("loop done") # the generator is paused, not closed
gen.close()
When cleanup must happen at a known point, close it yourself, or use with contextlib.closing(gen) as g:.
itertools
itertools is a module of fast, lazy building blocks. Most "loop with an index and a temporary list" code has
a one-line itertools version. All of these return iterators.
| Function | What it does | Example | Result |
|---|---|---|---|
count(start, step) | counts forever | count(10, 5) | 10, 15, 20, ... |
cycle(xs) | repeats xs forever | cycle("AB") | A, B, A, B, ... |
repeat(x, n) | x, n times (or forever) | repeat(0, 3) | 0, 0, 0 |
islice(xs, start, stop, step) | slices any iterator | islice(count(), 2, 8, 2) | 2, 4, 6 |
chain(a, b) / chain.from_iterable(xss) | concatenates / flattens one level | chain([1], (2, 3)) | 1, 2, 3 |
accumulate(xs, fn) | running totals (or any running fold) | accumulate([1, 2, 3]) | 1, 3, 6 |
pairwise(xs) | overlapping pairs | pairwise("abc") | (a, b), (b, c) |
batched(xs, n) | tuples of n (3.12+) | batched("abcde", 2) | (a, b), (c, d), (e,) |
groupby(xs, key) | runs of equal keys | groupby("aabca") | a, b, c, a |
takewhile(p, xs) / dropwhile(p, xs) | the prefix where p holds / the rest | takewhile(lambda n: n < 3, [1, 2, 5, 1]) | 1, 2 |
product(a, b) | every combination, like nested loops | product("ab", [1, 2]) | (a, 1), (a, 2), (b, 1), (b, 2) |
permutations(xs, r) | ordered arrangements | permutations("abc", 2) | 6 tuples |
combinations(xs, r) | unordered selections | combinations("abcd", 2) | 6 tuples |
tee(xs, n) | n independent iterators over one | a, b = tee(it) | |
zip_longest(a, b) | zip that pads the shorter one | zip_longest("ab", "c") | (a, c), (b, None) |
Infinite sources and slicing
from itertools import chain, count, cycle, islice
print(list(islice(count(10, 5), 4)))
print(list(islice(cycle("AB"), 5)))
print(list(chain([1, 2], (3, 4), "ab")))
print(list(chain.from_iterable([[1], [2, 3], []])))
print(list(islice(range(100), 2, 10, 3)))
islice on an iterator consumes what it takes, so calling it repeatedly walks through the data in chunks:
from itertools import islice
it = iter(range(10))
print(list(islice(it, 3)))
print(list(islice(it, 3)))
cycle is handy for round-robin assignment: zip(tickets, cycle(agents)) deals tickets to agents in turn.
Running totals, neighbours and batches
import operator
from itertools import accumulate, batched, pairwise
moves = [100, -30, 50, -20]
print(list(accumulate(moves))) # running balance
print(list(accumulate(moves, initial=1000))) # starting from 1000
print(list(accumulate(moves, max))) # running maximum
print(list(accumulate([1, 2, 3, 4], operator.mul)))
readings = [3, 5, 4, 9]
print([b - a for a, b in pairwise(readings)]) # change between neighbours
print(list(batched(range(7), 3))) # tuples; the last one may be short
pairwise replaces for i in range(1, len(xs)): xs[i] - xs[i - 1], and batched replaces the slicing loop
for "send 100 rows per request". Both work on iterators, not only on lists.
groupby and the sorted-input trap
groupby groups consecutive items with the same key. It's the Unix uniq command, not SQL's
GROUP BY and not LINQ's GroupBy, which both collect every item with the key wherever it is:
from itertools import groupby
orders = [("mon", 5), ("tue", 3), ("mon", 2)]
day = lambda order: order[0]
for key, group in groupby(orders, key=day):
print(key, [qty for _, qty in group])
print("sorted first:")
for key, group in groupby(sorted(orders, key=day), key=day):
print(key, [qty for _, qty in group])
Unsorted, mon shows up as two groups. Sort by the same key first (or use a defaultdict(list) when you
don't need the ordering).
Second trap: each group is a view into the shared underlying iterator. Advancing groupby to the next key
uses up the previous group, so storing groups for later gives empty results:
from itertools import groupby
words = ["apple", "avocado", "banana", "blueberry", "cherry"]
saved = [(letter, group) for letter, group in groupby(words, key=lambda w: w[0])]
print([(letter, list(group)) for letter, group in saved])
fixed = [(letter, list(group)) for letter, group in groupby(words, key=lambda w: w[0])]
print(fixed)
Consume each group (with list, sum, len(list(...))) inside the loop.
takewhile and dropwhile
from itertools import dropwhile, takewhile
lines = ["# exported 2026-10-01", "# version 2", "id,total", "1,20", "# trailing note"]
print(list(takewhile(lambda l: l.startswith("#"), lines)))
print(list(dropwhile(lambda l: l.startswith("#"), lines)))
dropwhile only skips the leading run; once the predicate fails, everything after is yielded, including
the trailing comment. Also, takewhile has to pull the first failing item to know it should stop, and that
item is lost:
from itertools import takewhile
it = iter([1, 2, 10, 3])
print(list(takewhile(lambda n: n < 5, it)))
print(list(it)) # the 10 was consumed and dropped
Combinatorics
from itertools import combinations, permutations, product
print(list(product("ab", [1, 2])))
print(list(permutations("abc", 2)))
print(list(combinations("abcd", 2)))
print(len(list(product(range(10), repeat=3)))) # every 3-digit code
product replaces nested loops (a test matrix of browsers by locales), combinations gives every pair (compare
every two documents), and permutations gives orderings. They grow fast: permutations(range(10)) has
3,628,800 items, so iterate lazily and stop early instead of building a list.
tee
tee splits one iterator into several independent ones. It buffers the values that one copy has seen and the
others haven't yet:
from itertools import tee
readings = iter([3, 5, 4])
a, b = tee(readings)
print(list(a), list(b))
After calling tee, don't use the original iterator, or the copies miss values. If one copy will be consumed
completely before the other starts, tee buffers everything anyway, and list() is simpler and faster.
Lists or generators?
| Use a list when | Use a generator when |
|---|---|
You need len, indexing or slicing | The data is large, streaming, or infinite |
| You'll iterate more than once | You'll iterate once |
| The caller expects a concrete result to store | The consumer may stop early (any, next, islice) |
| The data is small and simplicity wins | You're building a pipeline of stages |
Interview checklist:
iter()gives a fresh iterator from an iterable; an iterator is one-shot.- A generator's body runs on
next(), not on the call. Validation inside it runs late. returnin a generator ends it; its value rides onStopIterationand comes back fromyield from.groupbyneeds sorted input; consume each group before moving on.breakdoesn't close a generator;close()orcontextlib.closingdoes.- Generator expressions evaluate their first iterable immediately and everything else lazily.
Practise this lesson
In the app: a short video, 7 exercises checked by real tests and a 12-question concept check. Free with a GitHub account.
Open in the app