Python · Lesson 8

Iterators and Generators

The iterator protocol behind every for loop, one-shot iterators, writing __iter__ and __next__, generator functions and yield, lazy and infinite sequences, yield from, generator expressions and pipelines, send and close, and the itertools functions that replace most hand-written loops.

17 min read

In short

  • iter() and next(); the end is StopIteration
  • Iterators are one-shot; lists are re-iterable
  • yield makes a lazy, pausable function
  • yield from delegates; genexps build pipelines
  • itertools: islice, pairwise, batched, groupby (sort first)

for loops, list(), sum(), max(), sorted(), in, unpacking (a, b = x), dict(zip(...)) and ", ".join(...) all work on lists, strings, dicts, files and ranges. None of them is special-cased for each type. They all speak one small protocol, and any object that speaks it works with all of them. Generators are the easy way to speak it, and itertools is the standard toolbox built on it.

If you know JavaScript, this is Symbol.iterator and function*. If you know C#, it's IEnumerable<T>, yield return and LINQ's deferred execution. The ideas carry over directly; the differences are in the details, and the details are what interviews ask about.

Every python snippet has a ▶ Try button. Predict the output first, then run it.

The protocol: iter and next

Two roles, two methods:

  • An iterable is anything you can loop over. It has __iter__(), which returns an iterator. The built-in iter(x) calls it.
  • An iterator produces the values. It has __next__(), which returns the next value or raises StopIteration when there are no more. The built-in next(it) calls it.
▶ Try
nums = [10, 20]
it = iter(nums)
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("StopIteration: no more values")
print(next(it, "done"))   # a default instead of the exception

A for loop is that protocol with the exception handled for you. for x in nums: print(x) does this:

▶ Try
nums = [10, 20]
it = iter(nums)
while True:
    try:
        x = next(it)
    except StopIteration:
        break
    print(x)

The same idea in the three languages:

PythonJavaScriptC#
Get an iteratoriter(x) calls x.__iter__()x[Symbol.iterator]()x.GetEnumerator()
Next valuenext(it) calls it.__next__()it.next() returns { value, done }it.MoveNext(), then it.Current
End of sequenceraises StopIterationdone: trueMoveNext() returns false
Generatora def containing yieldfunction* with yielda method with yield return
Delegate to anotheryield fromyield*foreach + yield return

Python signals the end with an exception. That sounds expensive, but for loops and built-ins handle it in C, and you rarely see it unless you call next() yourself.

Iterables vs iterators: one-shot

A list is iterable but not an iterator. Each iter(list) gives a fresh iterator starting at the beginning, so you can loop over a list as often as you like. An iterator's __iter__ returns itself, so it can be consumed only once:

▶ Try
nums = [1, 2, 3]
print(iter(nums) is iter(nums))   # two separate iterators
it = iter(nums)
print(iter(it) is it)             # an iterator is its own iterator
print(list(it))
print(list(it))                   # already used up

This is the most common iterator bug in real code: a function that walks its argument twice works with a list and silently breaks with a generator.

▶ Try
def average(values):
    total = sum(values)
    count = len(list(values))
    return total / count if count else None


print(average([3, 4, 5]))
print(average(v for v in [3, 4, 5]))   # sum() used everything up

The first call prints 4.0. The second prints None: sum consumed the generator, so list(values) was empty. If you need two passes, materialize once (values = list(values)) and say so in the type hint (Sequence instead of Iterable).

Many built-ins return iterators, not lists: map, filter, zip, enumerate, reversed, and file objects. They're lazy and one-shot. range is different: it's a lazy iterable that you can loop over again, measure with len, and test with in without looping.

▶ Try
squares = map(lambda n: n * n, [1, 2, 3])
print(type(squares).__name__)
print(list(squares), list(squares))

r = range(0, 10, 2)
print(list(r), list(r), len(r), 6 in r)

in on an iterator consumes it up to the match, which surprises people:

▶ Try
it = iter([1, 2, 3, 4])
print(3 in it)
print(list(it))   # 1, 2 and 3 are gone

Writing an iterator class

To make your own type iterable, give it __iter__. The classic textbook version splits the iterable from the iterator, so the iterable can be looped over many times:

▶ Try
class Countdown:
    def __init__(self, start: int) -> None:
        self.start = start

    def __iter__(self) -> "CountdownIterator":
        return CountdownIterator(self.start)


class CountdownIterator:
    def __init__(self, current: int) -> None:
        self.current = current

    def __iter__(self) -> "CountdownIterator":
        return self

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value


c = Countdown(3)
print(list(c), list(c))
for n in c:
    print(n, end=" ")
print()

In practice you rarely write the iterator class. Make __iter__ a generator (next section) and Python builds the iterator for you:

▶ Try
class Countdown:
    def __init__(self, start: int) -> None:
        self.start = start

    def __iter__(self):
        n = self.start
        while n > 0:
            yield n
            n -= 1


print(list(Countdown(3)), sum(Countdown(4)), max(Countdown(5)))

Write a real iterator class when the iterator needs more methods than next: a peek(), a push_back(), a .position for error messages. The Peekable exercise is that case.

One more built-in trick: iter(callable, sentinel) calls the function until it returns the sentinel. It turns "read until empty" loops into iterators:

▶ Try
import io

f = io.StringIO("abcdefgh")
for chunk in iter(lambda: f.read(3), ""):
    print(chunk)

Generators: functions that pause

A function that contains yield is a generator function. Calling it doesn't run the body. It returns a generator object, which is an iterator. Each next() runs the body until the next yield, hands out that value, and pauses there with all local variables intact.

▶ Try
def steps():
    print("  start")
    yield "a"
    print("  after a")
    yield "b"
    print("  after b")
    return "the end"


gen = steps()          # nothing printed: the body hasn't started
print("created")
print(next(gen))
print(next(gen))
try:
    next(gen)
except StopIteration as e:
    print("finished, return value:", e.value)
print(list(gen))       # finished generators stay finished

Things to notice:

  • Lazy. Code runs only when someone asks for the next value.
  • return ends the sequence. Its value travels on the StopIteration (e.value), and for, list() and friends ignore it. It's only useful with yield from (below).
  • One-shot. A generator object is an iterator, so it can be consumed once. Call the function again for a fresh one.

C# developers: this is yield return, and return in a generator is like yield break (plus an optional value). JavaScript developers: function* without the asterisk. Python decides that a function is a generator by the presence of yield anywhere in its body.

Laziness, step by step

Chained generators pass values along one at a time. Watch the order:

▶ Try
def numbers():
    for n in [1, 2, 3]:
        print("produce", n)
        yield n


def doubled(values):
    for v in values:
        print("  double", v)
        yield v * 2


for result in doubled(numbers()):
    print("    consume", result)

Each value travels the whole chain before the next one is produced: produce 1, double 1, consume 2, then produce 2... With list comprehensions instead, all three "produce" lines would print first. That's what makes generators work on data bigger than memory, and on data that never ends.

Infinite generators

Because nothing runs ahead, a generator can describe an endless sequence. The consumer decides how much to take, usually with itertools.islice or a break:

▶ Try
from itertools import islice


def ids(prefix: str):
    n = 1
    while True:
        yield f"{prefix}-{n:04d}"
        n += 1


print(list(islice(ids("INV"), 3)))

for invoice_id in ids("ORD"):
    if invoice_id.endswith("3"):
        break
print("stopped at", invoice_id)

Calling list() on an infinite generator never returns. In an interview, say how the consumer bounds it.

Memory

A list comprehension builds every element; a generator holds one at a time. That's the reason to choose one over the other for big inputs:

▶ Try
import sys

as_list = [n * n for n in range(100_000)]
as_gen = (n * n for n in range(100_000))
print(sys.getsizeof(as_list) > 100_000, sys.getsizeof(as_gen) < 1_000)
print(sum(as_gen) == sum(as_list))

The list takes hundreds of kilobytes (more on a 64-bit machine); the generator object is a couple of hundred bytes, however long the sequence.

Two traps

Validation in a generator runs late. The body, including any argument checks at the top, waits for the first next():

▶ Try
def chunks(items, size):
    if size < 1:
        raise ValueError("size must be at least 1")
    for i in range(0, len(items), size):
        yield items[i:i + size]


gen = chunks([1, 2, 3], 0)
print("no error yet")
try:
    next(gen)
except ValueError as e:
    print("only now:", e)

The fix is a plain function that validates and then returns an inner generator. The Message batches exercise uses it.

StopIteration inside a generator becomes RuntimeError. A bare next() that hits the end inside a generator would silently end the outer generator, hiding the bug. Since Python 3.7 (PEP 479) it raises instead:

▶ Try
def first_of_each(groups):
    for group in groups:
        yield next(iter(group))   # an empty group raises StopIteration here


try:
    print(list(first_of_each([[1, 2], [], [3]])))
except RuntimeError as e:
    print("RuntimeError:", e)

Use next(iter(group), None) with a default, or a for loop with break.

yield from

yield from iterable yields every value of another iterable, as if you'd written for x in iterable: yield x. It's how a generator delegates, and recursion becomes natural:

▶ Try
def flatten(items):
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item


print(list(flatten([1, [2, [3, 4]], [], 5])))

Walking a tree is the same shape:

▶ Try
tree = {"name": "src", "children": [
    {"name": "app.py", "children": []},
    {"name": "lib", "children": [{"name": "util.py", "children": []}]},
]}


def walk(node, depth=0):
    yield "  " * depth + node["name"]
    for child in node["children"]:
        yield from walk(child, depth + 1)


print("\n".join(walk(tree)))

yield from does more than the loop: it passes send() and close() through to the inner generator, and it evaluates to the inner generator's return value:

▶ Try
def inner():
    yield 1
    yield 2
    return "inner done"


def outer():
    result = yield from inner()
    print("inner returned:", result)
    yield 3


print(list(outer()))

This is JavaScript's yield*. C# has no equivalent; you write foreach (var x in Inner()) yield return x;.

Generator expressions

A generator expression is a list comprehension in parentheses: (expr for x in xs if cond). It's lazy. When it's the only argument of a call, the extra parentheses can go:

▶ Try
orders = [120, 80, 300, 45]
print(sum(o for o in orders if o >= 100))
print(max(len(word) for word in ["iter", "generator", "yield"]))
print(", ".join(str(o) for o in orders))

Laziness matters with any and all, which stop at the first answer they can give:

▶ Try
def is_big(n):
    print("checking", n)
    return n > 2


print(any(is_big(n) for n in [1, 3, 5, 7]))
print(any([is_big(n) for n in [1, 3, 5, 7]]))

The generator version checks 1 and 3 and stops. The list version builds the whole list first, checking all four, and only then asks any.

One subtle rule: the first for's iterable is evaluated immediately, when the generator expression is created. Everything else (the expression, the conditions) runs later, as values are pulled:

▶ Try
nums = [1, 2, 3]
factor = 10
gen = (n * factor for n in nums)
nums = [7, 8, 9]       # too late: the generator already holds the old list
factor = 100           # not too late: factor is read as each value is made
print(list(gen))

It prints [100, 200, 300]: the old list's values, times the new factor. The same rule explains why a generator expression over a list you later mutate sees the mutation (it holds the list object, not a copy).

Pipelines

Generators compose into pipelines, like Unix pipes: each stage pulls from the previous one, and nothing is stored in between.

▶ Try
from collections import Counter

log = """\
INFO start
ERROR db timeout
INFO retry
ERROR db timeout

WARN slow query
ERROR disk full
"""

lines = (line.strip() for line in log.splitlines())
entries = (line.partition(" ") for line in lines if line)
errors = (message for level, _, message in entries if level == "ERROR")
print(Counter(errors).most_common(2))

None of the first three lines does any work. Counter(...) pulls from errors, which pulls from entries, which pulls from lines. With a real file it's the same code with open(path) as the source (a file object is an iterator of lines), and it runs in constant memory on a 10 GB log. Keep the whole pipeline inside the with open(...) block, because the generators read from the file lazily.

This is C#'s LINQ (lines.Where(...).Select(...), deferred until enumerated) and JavaScript's iterator helpers (Iterator.from(x).filter(...).map(...)). Python writes it with generator expressions and functions instead of method chains.

send and close, briefly

A generator can also receive values. gen.send(value) resumes it and makes the paused yield expression evaluate to value. You must start it with next() first, to run it to the first yield:

▶ Try
def running_average():
    total = 0.0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count


avg = running_average()
next(avg)              # run to the first yield
print(avg.send(10))
print(avg.send(20))
print(avg.send(60))

This was the basis of Python's early coroutines. Today async def covers that use, and send is rare in application code. You should recognise it, not reach for it.

gen.close() raises GeneratorExit at the paused yield, so finally blocks (and with blocks) inside the generator run. That's how a generator that holds a resource cleans up when the consumer stops early:

▶ Try
def numbered(lines):
    print("open")
    try:
        for i, line in enumerate(lines, 1):
            yield f"{i}: {line}"
    finally:
        print("cleanup")


gen = numbered(["a", "b", "c"])
print(next(gen))
gen.close()
print("after close")

Unlike JavaScript's for...of, a Python for loop doesn't close the iterator when you break. CPython closes a generator when the last reference to it disappears, which is usually right away, but not when you keep the generator in a variable:

▶ Try
def source():
    try:
        yield 1
        yield 2
    finally:
        print("source closed")


gen = source()
for x in gen:
    print("got", x)
    break
print("loop done")     # the generator is paused, not closed
gen.close()

When cleanup must happen at a known point, close it yourself, or use with contextlib.closing(gen) as g:.

itertools

itertools is a module of fast, lazy building blocks. Most "loop with an index and a temporary list" code has a one-line itertools version. All of these return iterators.

FunctionWhat it doesExampleResult
count(start, step)counts forevercount(10, 5)10, 15, 20, ...
cycle(xs)repeats xs forevercycle("AB")A, B, A, B, ...
repeat(x, n)x, n times (or forever)repeat(0, 3)0, 0, 0
islice(xs, start, stop, step)slices any iteratorislice(count(), 2, 8, 2)2, 4, 6
chain(a, b) / chain.from_iterable(xss)concatenates / flattens one levelchain([1], (2, 3))1, 2, 3
accumulate(xs, fn)running totals (or any running fold)accumulate([1, 2, 3])1, 3, 6
pairwise(xs)overlapping pairspairwise("abc")(a, b), (b, c)
batched(xs, n)tuples of n (3.12+)batched("abcde", 2)(a, b), (c, d), (e,)
groupby(xs, key)runs of equal keysgroupby("aabca")a, b, c, a
takewhile(p, xs) / dropwhile(p, xs)the prefix where p holds / the resttakewhile(lambda n: n < 3, [1, 2, 5, 1])1, 2
product(a, b)every combination, like nested loopsproduct("ab", [1, 2])(a, 1), (a, 2), (b, 1), (b, 2)
permutations(xs, r)ordered arrangementspermutations("abc", 2)6 tuples
combinations(xs, r)unordered selectionscombinations("abcd", 2)6 tuples
tee(xs, n)n independent iterators over onea, b = tee(it)
zip_longest(a, b)zip that pads the shorter onezip_longest("ab", "c")(a, c), (b, None)

Infinite sources and slicing

▶ Try
from itertools import chain, count, cycle, islice

print(list(islice(count(10, 5), 4)))
print(list(islice(cycle("AB"), 5)))
print(list(chain([1, 2], (3, 4), "ab")))
print(list(chain.from_iterable([[1], [2, 3], []])))
print(list(islice(range(100), 2, 10, 3)))

islice on an iterator consumes what it takes, so calling it repeatedly walks through the data in chunks:

▶ Try
from itertools import islice

it = iter(range(10))
print(list(islice(it, 3)))
print(list(islice(it, 3)))

cycle is handy for round-robin assignment: zip(tickets, cycle(agents)) deals tickets to agents in turn.

Running totals, neighbours and batches

▶ Try
import operator
from itertools import accumulate, batched, pairwise

moves = [100, -30, 50, -20]
print(list(accumulate(moves)))                 # running balance
print(list(accumulate(moves, initial=1000)))   # starting from 1000
print(list(accumulate(moves, max)))            # running maximum
print(list(accumulate([1, 2, 3, 4], operator.mul)))

readings = [3, 5, 4, 9]
print([b - a for a, b in pairwise(readings)])  # change between neighbours

print(list(batched(range(7), 3)))              # tuples; the last one may be short

pairwise replaces for i in range(1, len(xs)): xs[i] - xs[i - 1], and batched replaces the slicing loop for "send 100 rows per request". Both work on iterators, not only on lists.

groupby and the sorted-input trap

groupby groups consecutive items with the same key. It's the Unix uniq command, not SQL's GROUP BY and not LINQ's GroupBy, which both collect every item with the key wherever it is:

▶ Try
from itertools import groupby

orders = [("mon", 5), ("tue", 3), ("mon", 2)]
day = lambda order: order[0]

for key, group in groupby(orders, key=day):
    print(key, [qty for _, qty in group])

print("sorted first:")
for key, group in groupby(sorted(orders, key=day), key=day):
    print(key, [qty for _, qty in group])

Unsorted, mon shows up as two groups. Sort by the same key first (or use a defaultdict(list) when you don't need the ordering).

Second trap: each group is a view into the shared underlying iterator. Advancing groupby to the next key uses up the previous group, so storing groups for later gives empty results:

▶ Try
from itertools import groupby

words = ["apple", "avocado", "banana", "blueberry", "cherry"]
saved = [(letter, group) for letter, group in groupby(words, key=lambda w: w[0])]
print([(letter, list(group)) for letter, group in saved])

fixed = [(letter, list(group)) for letter, group in groupby(words, key=lambda w: w[0])]
print(fixed)

Consume each group (with list, sum, len(list(...))) inside the loop.

takewhile and dropwhile

▶ Try
from itertools import dropwhile, takewhile

lines = ["# exported 2026-10-01", "# version 2", "id,total", "1,20", "# trailing note"]
print(list(takewhile(lambda l: l.startswith("#"), lines)))
print(list(dropwhile(lambda l: l.startswith("#"), lines)))

dropwhile only skips the leading run; once the predicate fails, everything after is yielded, including the trailing comment. Also, takewhile has to pull the first failing item to know it should stop, and that item is lost:

▶ Try
from itertools import takewhile

it = iter([1, 2, 10, 3])
print(list(takewhile(lambda n: n < 5, it)))
print(list(it))    # the 10 was consumed and dropped

Combinatorics

▶ Try
from itertools import combinations, permutations, product

print(list(product("ab", [1, 2])))
print(list(permutations("abc", 2)))
print(list(combinations("abcd", 2)))
print(len(list(product(range(10), repeat=3))))   # every 3-digit code

product replaces nested loops (a test matrix of browsers by locales), combinations gives every pair (compare every two documents), and permutations gives orderings. They grow fast: permutations(range(10)) has 3,628,800 items, so iterate lazily and stop early instead of building a list.

tee

tee splits one iterator into several independent ones. It buffers the values that one copy has seen and the others haven't yet:

▶ Try
from itertools import tee

readings = iter([3, 5, 4])
a, b = tee(readings)
print(list(a), list(b))

After calling tee, don't use the original iterator, or the copies miss values. If one copy will be consumed completely before the other starts, tee buffers everything anyway, and list() is simpler and faster.

Lists or generators?

Use a list whenUse a generator when
You need len, indexing or slicingThe data is large, streaming, or infinite
You'll iterate more than onceYou'll iterate once
The caller expects a concrete result to storeThe consumer may stop early (any, next, islice)
The data is small and simplicity winsYou're building a pipeline of stages

Interview checklist:

  • iter() gives a fresh iterator from an iterable; an iterator is one-shot.
  • A generator's body runs on next(), not on the call. Validation inside it runs late.
  • return in a generator ends it; its value rides on StopIteration and comes back from yield from.
  • groupby needs sorted input; consume each group before moving on.
  • break doesn't close a generator; close() or contextlib.closing does.
  • Generator expressions evaluate their first iterable immediately and everything else lazily.

Practise this lesson

In the app: a short video, 7 exercises checked by real tests and a 12-question concept check. Free with a GitHub account.

Open in the app