Python: Iterators and Generators
Last updated: 2026-09-22
You've been using
for x in list:to iterate data, but have you ever thought about what can followin? Why doesn'trange(1000000)take up memory? Iterators and generators are the answer. They let you handle "infinite" data streams — generated on demand, not stored in memory.
1. What Is the Iterator Protocol
In Python, any object that can be used in a for loop is iterable. They implement the iterator protocol:
▶ Example: Iterator Protocol Basics
▶ Try it YourselfOutput:
# Executed successfully
A for loop essentially does this:
▶ Example: Simulating a for Loop Manually
▶ Try it YourselfOutput:
# Executed successfully
2. Custom Iterators
Implement __iter__ and __next__ to make your class work with for loops:
▶ Example: Countdown Iterator
▶ Try it YourselfOutput:
# Function defined successfully
▶ Example: Fibonacci Iterator (Difficulty ⭐⭐)
▶ Try it YourselfOutput:
# Function defined successfully
3. Generators: yield
Writing a class with __iter__ and __next__ is tedious. Generator functions use yield to achieve the same thing:
▶ Example: Countdown Generator
▶ Try it YourselfOutput:
# Function defined successfully
yield is different from return — return ends a function, while yield pauses it, remembers the current state, and resumes from where it left off on the next call.
▶ Example: Fibonacci Generator
▶ Try it YourselfOutput:
# Function defined successfully
4. Lazy Loading with Generators
The biggest advantage of generators — they don't generate all data at once, but on demand:
▶ Example: Generator Lazy Loading
▶ Try it YourselfOutput:
Generating 1
Generating 2
Generating 3
Generator created, but not yet executed
See the execution flow? Each call to next() executes up to the next yield, then pauses. This is incredibly useful for large data:
▶ Example: Reading Large Files Line by Line
▶ Try it YourselfOutput:
# Function defined successfully
5. Generator Expressions
Similar to list comprehensions, but with parentheses — generator expressions are lazily evaluated:
▶ Example: Generator Expression
▶ Try it YourselfOutput:
# Executed successfully
| Comparison | List Comprehension [] |
Generator Expression () |
|---|---|---|
| Memory | All data in memory | On demand, almost none |
| Speed | Fast generation and access | Lazy, computes each time |
| Reusable | Can iterate multiple times | Can only iterate once |
| Use case | Small data, repeated use | Large data, single pass |
6. Common Use Cases
- Big data processing: Read large log/CSV files line by line, avoiding OOM.
- Infinite sequences: Generators can represent "infinite" data streams — Fibonacci, sensor data.
- Pipeline processing: Chain multiple generators into a data processing pipeline.
- Reverse iteration:
reversed()returns an iterator for some types. - Lazy evaluation: Functions return a generator; the caller decides how many values to take.
❓ FAQ
return to return a result once and ends. A generator uses yield to produce values one at a time, pausing and saving state after each yield. Each call to a generator function returns a new generator object.list(gen) to convert the generator to a list. But if the data is too large, converting to a list will exhaust memory. A compromise: use itertools.tee() to copy the generator, or call the generator function again to create a new one.yield and return coexist?return in a generator effectively stops iteration and raises StopIteration with an optional value. But it's rarely used. If you find a function needing both yield and return, reconsider the design.📖 Summary
- Iterable objects support
forloops; behind the scenes, they implement the__iter__and__next__protocol - Manual iteration:
iter()gets an iterator,next()gets the next value,StopIterationends it - Generator functions use
yieldto produce values one at a time, auto-saving state - Generators are lazy — produce as much as needed, no memory waste
- Generator expressions
(x for x in seq)are like list comprehensions, but lazily evaluated yield fromdelegates to another generator (advanced, learn when you need it)
📝 Exercises
-
Basic (Difficulty ⭐): Write a generator
even_numbers(max_n)that generates all even numbers from 0 tomax_n. -
Intermediate (Difficulty ⭐⭐): Write a generator
repeat_list(items, times)that repeats each element of a listtimestimes. For example,list(repeat_list([1, 2, 3], 2))returns[1, 1, 2, 2, 3, 3]. -
Challenge (Difficulty ⭐⭐⭐): Write a generator
deep_flatten(nested)that flattens arbitrarily nested lists. For example,list(deep_flatten([1, [2, [3, 4]], 5]))returns[1, 2, 3, 4, 5]. Hint: Iterate over elements; if an element is a list, recursively yield from it withyield from.
