Iterators and Generators
You've been using
for x in list:to iterate data, but have you ever thought about what can followin? Why doesn'trange(1000000)take up memory? Iterators and generators are the answer. They let you handle "infinite" data streams — generated on demand, not stored in memory.
1. What Is the Iterator Protocol
In Python, any object that can be used in a for loop is iterable. They implement the iterator protocol:
Example: Iterator Protocol Basics
# All containers are iterable
print(hasattr([1, 2, 3], "__iter__")) # True
print(hasattr("abc", "__iter__")) # True
print(hasattr(100, "__iter__")) # False — numbers are not iterable
# Manual iteration process
numbers = [1, 2, 3]
iterator = iter(numbers) # Get an iterator
print(next(iterator)) # 1
print(next(iterator)) # 2
print(next(iterator)) # 3
# print(next(iterator)) # StopIteration — iteration complete
A for loop essentially does this:
Example: Simulating a for Loop Manually
numbers = [1, 2, 3]
it = iter(numbers)
while True:
try:
value = next(it)
print(value)
except StopIteration:
break
2. Custom Iterators
Implement __iter__ and __next__ to make your class work with for loops:
Example: Countdown Iterator
class Countdown:
"""Countdown iterator"""
def __init__(self, start):
self.current = start
def __iter__(self):
return self # Iterator returns itself
def __next__(self):
if self.current <= 0:
raise StopIteration # Stop iteration
value = self.current
self.current -= 1
return value
# Usage
for i in Countdown(5):
print(i, end=" ") # 5 4 3 2 1
Example: Fibonacci Iterator (Difficulty ⭐⭐)
class Fibonacci:
"""Fibonacci iterator — generates the first N numbers"""
def __init__(self, count):
self.count = count
self.index = 0
self.a, self.b = 0, 1
def __iter__(self):
return self
def __next__(self):
if self.index >= self.count:
raise StopIteration
self.index += 1
result = self.a
self.a, self.b = self.b, self.a + self.b
return result
for num in Fibonacci(10):
print(num, end=" ") # 0 1 1 2 3 5 8 13 21 34
3. Generators: yield
Writing a class with __iter__ and __next__ is tedious. Generator functions use yield to achieve the same thing:
Example: Countdown Generator
def countdown(start):
"""Countdown generator"""
while start > 0:
yield start
start -= 1
# Usage
for i in countdown(5):
print(i, end=" ") # 5 4 3 2 1
yield is different from return — return ends a function, while yield pauses it, remembers the current state, and resumes from where it left off on the next call.
Example: Fibonacci Generator
# Same Fibonacci — using yield, much more concise than the iterator class
def fibonacci(count):
a, b = 0, 1
for _ in range(count):
yield a
a, b = b, a + b
for num in fibonacci(10):
print(num, end=" ") # 0 1 1 2 3 5 8 13 21 34
4. Lazy Loading with Generators
The biggest advantage of generators — they don't generate all data at once, but on demand:
Example: Generator Lazy Loading
def generate_numbers():
"""Simulate generating numbers one by one"""
print("Generating 1")
yield 1
print("Generating 2")
yield 2
print("Generating 3")
yield 3
gen = generate_numbers()
print("Generator created, but not yet executed")
print(next(gen)) # Generating 1 \n 1
print(next(gen)) # Generating 2 \n 2
print(next(gen)) # Generating 3 \n 3
See the execution flow? Each call to next() executes up to the next yield, then pauses. This is incredibly useful for large data:
Example: Reading Large Files Line by Line
# "Lazy" way to process large files
def read_large_file(filename):
"""Read a large file line by line, without loading it all into memory"""
with open(filename, "r", encoding="utf-8") as f:
for line in f:
yield line.strip()
# Process with generator — only one line in memory at a time
for line in read_large_file("huge_log.txt"):
if "ERROR" in line:
print(line) # Speed is only limited by disk read speed
5. Generator Expressions
Similar to list comprehensions, but with parentheses — generator expressions are lazily evaluated:
Example: Generator Expression
# List comprehension — generates all data at once
squares_list = [x ** 2 for x in range(1000000)]
print(f"List size: {len(squares_list)}") # 1000000 (uses memory)
# Generator expression — on demand
squares_gen = (x ** 2 for x in range(1000000))
print(f"Generator: {squares_gen}") # generator object (uses almost no memory)
# Same usage — for loop
for i, val in enumerate(squares_gen):
if i >= 5:
break
print(val, end=" ") # 0 1 4 9 16
| Comparison | List Comprehension [] |
Generator Expression () |
|---|---|---|
| Memory | All data in memory | On demand, almost none |
| Speed | Fast generation and access | Lazy, computes each time |
| Reusable | Can iterate multiple times | Can only iterate once |
| Use case | Small data, repeated use | Large data, single pass |
Common Use Cases
- Big data processing: Read large log/CSV files line by line, avoiding OOM.
- Infinite sequences: Generators can represent "infinite" data streams — Fibonacci, sensor data.
- Pipeline processing: Chain multiple generators into a data processing pipeline.
- Reverse iteration:
reversed()returns an iterator for some types. - Lazy evaluation: Functions return a generator; the caller decides how many values to take.
❓ FAQ
return to return a result once and ends. A generator uses yield to produce values one at a time, pausing and saving state after each yield. Each call to a generator function returns a new generator object.
Q: Generators can only be iterated once. What if I need to iterate multiple times? A: Use list(gen) to convert the generator to a list. But if the data is too large, converting to a list will exhaust memory. A compromise: use itertools.tee() to copy the generator, or call the generator function again to create a new one.
Q: Can yield and return coexist? A: Yes — return in a generator effectively stops iteration and raises StopIteration with an optional value. But it's rarely used. If you find a function needing both yield and return, reconsider the design.📖 Summary
- Iterable objects support
forloops; behind the scenes, they implement the__iter__and__next__protocol - Manual iteration:
iter()gets an iterator,next()gets the next value,StopIterationends it - Generator functions use
yieldto produce values one at a time, auto-saving state - Generators are lazy — produce as much as needed, no memory waste
- Generator expressions
(x for x in seq)are like list comprehensions, but lazily evaluated yield fromdelegates to another generator (advanced, learn when you need it)
📝 Exercises
-
Basic (Difficulty ⭐): Write a generator
even_numbers(max_n)that generates all even numbers from 0 tomax_n. -
Intermediate (Difficulty ⭐⭐): Write a generator
repeat_list(items, times)that repeats each element of a listtimestimes. For example,list(repeat_list([1, 2, 3], 2))returns[1, 1, 2, 2, 3, 3]. -
Challenge (Difficulty ⭐⭐⭐): Write a generator
deep_flatten(nested)that flattens arbitrarily nested lists. For example,list(deep_flatten([1, [2, [3, 4]], 5]))returns[1, 2, 3, 4, 5]. Hint: Iterate over elements; if an element is a list, recursively yield from it withyield from.