Python · Lesson 3

Comprehensions and Iteration

List, dict and set comprehensions, generator expressions, enumerate and zip, sorting with key functions and stable multi-key sorts, any and all, sum, map and filter, and when a plain loop reads better.

11 min read

In short

  • Comprehensions: expression, for, optional if
  • Round brackets are lazy and single-use
  • enumerate, and zip(..., strict=True)
  • sorted with a key; tuples for several keys; stable
  • any, all, sum; a loop for side effects

Most code that handles data walks over a collection and builds another one: filter the active users, map rows to ids, total the invoice lines, sort by date. JavaScript does it with .filter().map() chains, C# with LINQ. Python's answer is the comprehension, plus a small set of built-ins (enumerate, zip, sorted, any, all, sum) that take any iterable. Together they replace most hand-written loops, and interviewers expect you to reach for them.

You wantJavaScriptC# LINQPython
transform eachxs.map(f)xs.Select(f)[f(x) for x in xs]
keep somexs.filter(p)xs.Where(p)[x for x in xs if p(x)]
index and itemxs.forEach((x, i) => ...)xs.Select((x, i) => ...)for i, x in enumerate(xs)
pair two listsn/aa.Zip(b)zip(a, b)
any / everysome / everyAny / Allany(...) / all(...)
totalreduceSumsum(...)
sort bysort((a, b) => ...)OrderBy(k).ThenBy(k2)sorted(xs, key=...)

Every snippet has a ▶ Try button. Predict the output, then run it.

List comprehensions

A list comprehension builds a list from an iterable: an expression, a for clause, and an optional if filter. Read it as "this expression, for each x, if the condition holds":

▶ Try
prices = [12, 7, 30, 4]
print([p * 2 for p in prices])
print([p for p in prices if p >= 10])
print([f"${p}" for p in prices if p % 2 == 0])

The filter if goes at the end and drops items. A conditional expression goes at the front and picks a value for every item. They're easy to mix up:

▶ Try
nums = [3, -1, 4, -5]
print([n for n in nums if n > 0])          # filter: fewer items
print([n if n > 0 else 0 for n in nums])   # choose: same number of items

A comprehension is an expression, so it goes straight into a return, an argument or a dict value. Its loop variable stays inside it: after [x for x in xs], there's no x in the surrounding scope.

Dict and set comprehensions

The same shape with curly braces builds a dict (with key: value) or a set:

▶ Try
users = [("u1", "ann"), ("u2", "bo"), ("u3", "ann")]
by_id = {uid: name for uid, name in users}
names = {name for _, name in users}
lengths = {name: len(name) for name in sorted(names)}
print(by_id)
print(sorted(names), lengths)

Inverting a dict is a one-liner, and a classic interview question. When two keys share a value, the last one wins, because a later assignment to the same key replaces the earlier one:

▶ Try
role = {"ann": "admin", "bo": "dev", "cy": "admin"}
print({r: name for name, r in role.items()})

If you need every key per value, that's grouping, and a loop with setdefault is the clearer tool (see the Collections topic).

Nested comprehensions

Several for clauses nest in the order you'd write the loops, outermost first. That's the rule to remember, because it reads backwards to many people:

▶ Try
orders = [["pen", "ink"], [], ["mug"]]
items = [item for order in orders for item in order]
print(items)

# The same as:
items = []
for order in orders:
    for item in order:
        items.append(item)
print(items)

A comprehension inside the expression builds nested lists, like rows of a grid. Transposing a matrix uses one:

▶ Try
matrix = [[1, 2, 3], [4, 5, 6]]
print([[row[c] for row in matrix] for c in range(3)])
print([(r, c) for r in range(2) for c in range(2) if r != c])

Two for clauses are fine. Three, or two with conditions on each, is usually the point where a named helper function or a plain loop reads better.

Generator expressions: lazy comprehensions

Parentheses instead of brackets make a generator expression. It produces items one at a time, when something asks for them, instead of building a list. Passed straight to a function, it doesn't even need its own parentheses:

▶ Try
nums = range(1, 1_000_001)
print(sum(n * n for n in nums))         # no million-item list is built
print(max(len(w) for w in ["a", "abc", "ab"]))

squares = (n * n for n in [1, 2, 3])
print(squares)                           # a generator object, not the values
print(list(squares))
print(list(squares))                     # already used up

A generator can be consumed once. The second list(squares) is empty, without an error. That's the same as a C# IEnumerable built with yield return, except that C# reruns the query on each enumeration and Python's generator is simply exhausted. Pass a list when a function needs to walk the data twice.

next(gen, default) takes the first item and stops there. With a generator expression it's the idiomatic "find the first match", like JavaScript's find:

▶ Try
users = [{"name": "ann", "admin": False}, {"name": "bo", "admin": True}, {"name": "cy", "admin": True}]
first_admin = next((u["name"] for u in users if u["admin"]), None)
nobody = next((u["name"] for u in users if u["name"] == "zed"), None)
print(first_admin, nobody)

Without the default, next raises StopIteration when nothing matches.

enumerate and zip

enumerate(xs, start=0) gives (index, item) pairs, so you never need range(len(xs)):

▶ Try
steps = ["build", "test", "deploy"]
for n, step in enumerate(steps, start=1):
    print(f"{n}. {step}")

zip(a, b, ...) walks several iterables in step and yields tuples. It stops at the shortest input, silently, which hides bugs when two lists were supposed to match. strict=True (Python 3.10+) raises ValueError instead:

▶ Try
names = ["ann", "bo", "cy"]
scores = [90, 72]
print(list(zip(names, scores)))
print(dict(zip(["id", "name"], [7, "ann"])))
try:
    list(zip(names, scores, strict=True))
except ValueError as e:
    print("ValueError:", e)

zip(*rows) "unzips": the star spreads the rows as separate arguments, so zip pairs up their first items, then their second items. That's a transpose in one line. And zip(xs, xs[1:]) pairs each item with the next, which is how you compute differences between neighbours:

▶ Try
pairs = [("ann", 90), ("bo", 72), ("cy", 85)]
names, scores = zip(*pairs)
print(names, scores)

times = [0, 4, 5, 12]
print([b - a for a, b in zip(times, times[1:])])

sorted, min and max with a key

sorted(iterable, key=..., reverse=...) returns a new list. list.sort() takes the same arguments and sorts in place. Both take a key function that maps each item to the value to compare. JavaScript and C#'s Comparison<T> use a comparator (a, b) => number instead. A key is simpler: you say what to sort by, not how to compare two items.

▶ Try
words = ["banana", "Cherry", "apple", "fig"]
print(sorted(words))                    # uppercase sorts before lowercase
print(sorted(words, key=str.lower))
print(sorted(words, key=len, reverse=True))

min and max take the same key and return the item, not the key. Both raise ValueError on an empty iterable unless you pass default=:

▶ Try
products = [{"sku": "a", "price": 30}, {"sku": "b", "price": 12}, {"sku": "c", "price": 12}]
print(min(products, key=lambda p: p["price"]))
print(max([], default=None))

When several items tie, min and max return the first of them, which is why that prints product b.

Multi-key sorts and stability

A key that returns a tuple sorts by the first element, then the second for ties, and so on, like OrderBy(...).ThenBy(...). To sort one numeric part descending, negate it:

▶ Try
people = [("cy", 30), ("ann", 25), ("bo", 30), ("di", 25)]
print(sorted(people, key=lambda p: (p[1], p[0])))     # age, then name
print(sorted(people, key=lambda p: (-p[1], p[0])))    # age descending, then name

You can't negate a string. For mixed directions on strings, use the fact that Python's sort is stable: items that compare equal keep their original relative order. So sort by the least important key first, then by the more important ones:

▶ Try
tickets = [("bug", "bo"), ("feature", "ann"), ("bug", "ann"), ("chore", "cy")]
by_owner = sorted(tickets, key=lambda t: t[1])                  # secondary: owner A-Z
print(sorted(by_owner, key=lambda t: t[0], reverse=True))       # primary: kind Z-A

The final sort is by kind, and among the two bugs, the owner order from the first pass survives. reverse=True keeps stability too: it doesn't reverse the order of equal items. C#'s List.Sort is not stable (OrderBy is), and JavaScript's sort has been stable since ES2019.

reversed

reversed(seq) returns a lazy iterator that walks a sequence backwards without copying it. seq[::-1] builds a reversed copy. reversed needs a sequence (a length and indexes: a list, tuple, string or range) or an object with __reversed__, like a dict, so it fails on a generator or a set:

▶ Try
print(list(reversed([1, 2, 3])), "abc"[::-1], list(reversed(range(3))))
print(list(reversed({"a": 1, "b": 2})))      # dicts reverse their key order
try:
    reversed(n for n in [1, 2])
except TypeError as e:
    print("TypeError:", e)

any, all and sum

any is true if at least one item is truthy, and all if every item is. Both stop as soon as the answer is known, so with a generator expression they evaluate only as much as they need. On an empty iterable, any is False and all is True ("every one of zero items passed"), which matters when the list can be empty:

▶ Try
ports = [80, 443, 8080]
print(any(p > 1024 for p in ports), all(p > 0 for p in ports))
print(any([]), all([]))

def check(p: int) -> bool:
    print("checking", p)
    return p == 443

print(any(check(p) for p in ports))     # stops at 443

sum(iterable, start=0) adds numbers. The start value sets the starting point and the type: 0.0 for floats, Decimal("0") for money. It also concatenates lists, but that copies the growing list at every step, so it's quadratic. Flatten with a comprehension or itertools.chain instead. sum refuses strings outright, because "".join(...) is the right tool:

▶ Try
print(sum([1, 2, 3]), sum([1, 2, 3], start=10), sum([1, 2], 0.0))
print(sum([[1], [2, 3]], []))           # works, but O(n^2): avoid for big inputs
print(sum(p["qty"] * p["price"] for p in [{"qty": 2, "price": 5}, {"qty": 1, "price": 3}]))
try:
    sum(["a", "b"], "")
except TypeError as e:
    print("TypeError:", e)

Counting with sum is a common idiom: True is 1, so sum(x > 0 for x in xs) counts the positives.

map and filter versus comprehensions

map(f, xs) and filter(pred, xs) exist and return lazy iterators, like generator expressions. That surprises people from JavaScript, where map returns an array:

▶ Try
nums = ["3", "1", "2"]
print(map(int, nums))
print(list(map(int, nums)))
print(list(filter(None, ["a", "", "b", None, 0])))    # None as the predicate drops falsy items

Pythonic code prefers comprehensions in most cases, because they don't need a lambda and can map and filter in one step:

▶ Try
users = [{"name": "ann", "active": True}, {"name": "bo", "active": False}]
print(list(map(lambda u: u["name"], filter(lambda u: u["active"], users))))
print([u["name"] for u in users if u["active"]])

Both lines do the same; the second is what reviewers expect. map reads fine when you already have a named function: map(str, ids), map(int, line.split()).

When a plain loop is clearer

Comprehensions are for building a value. Use a plain for loop when:

  • The point is a side effect. [print(x) for x in xs] builds a list of Nones to throw away. Write the loop.
  • You build several results at once. Splitting into valid and invalid rows in one pass is two appends in a loop, not two comprehensions that each walk the data.
  • There's error handling or early exit with logic. A try/except can't go inside a comprehension.
  • It needs a third nesting level, or a condition you have to read twice. Name it.
▶ Try
rows = ["7", "x", "12", "", "-3"]
valid, errors = [], []
for line, raw in enumerate(rows, start=1):
    try:
        valid.append(int(raw))
    except ValueError:
        errors.append(f"line {line}: {raw!r} is not a number")
print(valid)
print(errors)

itertools, briefly

The itertools module has the building blocks for everything else, and they're all lazy:

FunctionWhat it does
chain(a, b) / chain.from_iterable(xss)one iterable after another; flattens one level
islice(it, n)the first n items of any iterable, even an endless one
groupby(xs, key)runs of consecutive items with the same key (sort first to group them all)
pairwise(xs)(x0, x1), (x1, x2)..., like zip(xs, xs[1:]) for any iterable
accumulate(xs)running totals
▶ Try
from itertools import accumulate, chain, groupby, islice, pairwise

print(list(chain([1, 2], [3])), list(islice("abcdef", 3)))
print([(k, len(list(g))) for k, g in groupby("aaabccaa")])
print(list(pairwise([1, 4, 9])), list(accumulate([5, 10, 20])))

groupby groups only consecutive items, so "aaabccaa" gives two separate runs of a. Sort by the same key first when you want one group per key.

The Iterators and Generators topic shows how all of this works underneath: the iterator protocol, yield, and writing your own lazy pipelines.

Practise this lesson

In the app: a short video, 7 exercises checked by real tests and a 12-question concept check. Free with a GitHub account.

Open in the app