Iterators and Generators in Python

Introduction

In python programming or any other programming language, looping over the sequence (or traversing) is the most common aspect. While loops and for loops are two loops in python that can handle most of the repeated tasks executed by programs. Iterating over sequences is so widespread that Python offers extra capabilities to make it easier and more efficient.

One of the tools for traversing is Iterators and Generators in Python. This chapter kicks off our investigation of these tools. Now, let’s get started with Iterators and Generators in python. 

Iterators in Python

In Python, an iterator is an object used to iterate over iterable objects such as lists, tuples, dictionaries, and sets. An object is called iterable if we can get an iterator from it or loop over it. 

Let us understand this with one example:

Suppose you have a list of even numbers:

Numbers = [2, 6, 8, 10, 12]

You can efficiently process the traversing using a For or While loop to get each element one by one.

They are neatly implemented within for loops, comprehensions, generators, etc., but concealed from view. In simple words, an iterator is just an object that can be iterated on.

A Python iterator object must implement two specific methods, __iter__() or iter() and __next__() or next() , which are referred to collectively as the iterator protocol.

Python iter()

The iter() function in Python returns an iterator for the supplied object. The iter() generates a thing that can be iterated one element at a time. These items are handy when combined with loops such as for loops and while loops. 

Syntax:

iter( object , sentinel )

iter() function takes two parameters:

• Object: An object whose iterator needs to be created (lists, sets, tuples, etc.).
• Sentinel (optional): Special value that represents the end of the sequence.

Python next()

The next() function returns the next item from the iterator. The next() function holds the value one at a time. 

Syntax:

next( iterator , default )

The next() method accepts two parameters:

• Iterator : next( ) function retrieves the next item from the iterator.
• default(optional): this value is returned if the iterator is exhausted (not tired, but no next item to retrieve).

Let’s consider an example for better understanding:

Assume we have a list of different types as given below. 

list1 = [ 25 , 78, ‘coding’, ‘is’, ‘<3’ ]  # list of different types

Let’s print it with the help of Iterators ( or iter() and next() ):-

# Program to print the list using Iterator protocols
X = [25, 78, 'Coding', 'is', '<3']
# Get an iterator using iter() 
a = iter(X)

# Printing the a iterator
print(a)

# next() for fetching the 1st element in the list that is 25
print(next(a))

# Fetch the 2nd element in the list that is 78
print(next(a))

# Fetching the consecutive elements
print(next(a))
print(next(a))
print(next(a))

Output

<list_iterator object at 0x000001B91F9CFFD0>
25
78
Coding
is
<3

As you can see, when we are trying to print the iterator a, it shows its type and the memory address it is located on. One by one, next() fetches the element from the list.

Let’s use the same example to figure out why there’s an exception at the end of the sequence in the above diagram. 

Let’s look at the next() method again in the above program to see what happens next.

# Trying to fetch the elements at the end of the sequence
print(next(a))

Output

<list_iterator object at 0x0000028A1D0EFFD0>
25
78
Coding
is
<3
Traceback (most recent call last):
  File "C:\Users\HP\PycharmProjects\pythonProject\main.py", line 15, in <module>
    print(next(a))
StopIteration

When we attempted to fetch the next value, we received an exception. Usually, a StopIteration Exception is raised when the next() method attempts to proceed to the next value, but there are no new values in the container.

What exactly is going on in the code’s background? Let’s have a look at the flowchart given below to understand it. 

Now, how to avoid the StopIteration Exception?

StopIteration is an iterator’s means of signalling that it has reached the end. When you use a for loop to iterate, the exception is handled internally and exploited to terminate the loop.

This is one of the distinctions between loops and iterators. When you explicitly call next(), you should be prepared to catch the exception yourself. A more elegant way to print the elements by using the loop is given below. 

We can wrap the code inside the try block, as shown below.

# Program to print the tuple using Iterator protocols
tup = (87, 90, 100, 500)

# get an iterator using iter()
tup_iter = iter(tup)

# Infinite loop
while True:
   try:
       # To fetch the next element
       print(next(tup_iter))
       # if exception is raised, break from the loop
   except StopIteration:
       break

Output

87
90
100
500

Let us consider some more examples:

Create your Iterator

Example 1: 

class MyNumbers:
 # __iter__() is same as iter()
  def __iter__(self):
   self.a = 1
   return self

  # __next__() is same as next()
  def __next__(self):
    # 20th is the highest value
    if self.a <= 5:
       x = self.a
      # Manually increment
       self.a += 1
      # returning the iterator to the function call
       return x

# Create the object of the class
myclass = MyNumbers()
# get an iterator using iter()
myiter = iter(myclass)

# printing the values using a for-in loop
for x in myiter:
 print(x)

Output

1
2
3
4
5
None
None
None
None
None
None
None
None
#------ goes on until control exit

After the maximum is hit, we will get the None an infinite number of times. The StopIteration statement can be used to halt the iteration from continuing indefinitely.

We can add a termination condition to the next() method to produce an error if the iteration is repeated a certain amount of times.

Example 2: 

class MyNumbers:
 # __iter__() is same as iter()
  def __iter__(self):
   self.a = 1
   return self

  # __next__() is same as next()
  def __next__(self):
    # 20th is the highest value
    if self.a <= 5:
       x = self.a
      # Manually increment
       self.a += 1
      # returning the iterator to the function call
       return x
      # added the terminating statement to prevent the iteration to go on forever
    else:
        raise StopIteration

# Create the object of the class
myclass = MyNumbers()
# get an iterator using iter()
myiter = iter(myclass)

# printing the values using a for-in loop
for x in myiter:
 print(x)

Output

1
2
3
4
5

Great, We made our iterator by defining its maximum length and including the terminating condition.

For a better understanding, consider the following example.

class PowerTwo:
   # Class to implement an iterator of powers of two
   # Constructor accepting the max value
   def __init__(self, max=0):
       self.max = max

   # defined __iter__() to point the first element
   def __iter__(self):
       self.n = 1
       return self

   # __next__() to fetch the next value from the iterator
   def __next__(self):
       if self.n <= self.max:
           result = 2 ** self.n
           self.n += 1
           return result

   # Terminating condition
       else:
           raise StopIteration


# create an object
numbers = PowerTwo(4)

# create an iterable from the object
i = iter(numbers)

# Using for-in loop to print the elements up to max
for it in i:
   print(it)

Output

2
4
8
16

In the above code, we are attempting to implement an iterator of powers of two manually. In the __iter__() method, we direct the n variable to the initial value. In the __next__() function, we manually increase the n variable to its maximum value. Simultaneously, the result is returned to an iterable object.

Generators in Python

Building an iterator in Python requires a significant amount of effort. We must create a class containing __iter__() and __next__() methods, keep track of internal states and raise StopIteration when no values are returned. This is both long and contradictory. In such cases, the generator comes to the rescue. 

Python has a generator that allows you to create your iterator function. A generator is somewhat of a function that returns an iterator object with a succession of values rather than a single item.

A yield statement, rather than a return statement, is used in a generator function. 

The difference is that, although a return statement terminates a function completely, a yield statement pauses the function while storing all of its states and then continues from there on subsequent calls.

Let’s retake the Power of two’s example to illustrate the concept. 

Example 1:

# Program to print the Power of two up to the given number
def PowerTwoGen( max=0 ):
   n = 1
   while n < max:
       yield 2 ** n
       n += 1

a = PowerTwoGen(6)

# Printing the values stored in a
for i in a:
   print(i)

Output

2
4
8
16
32

Since generators maintain track of information automatically, the implementation was shorter and more streamlined.

Let’s take another example for better comprehension:-

Example 2:

# A simple generator for Fibonacci Numbers
def fib(max):
   # Initialize first two Fibonacci Numbers
   p, q = 0, 1

   # yield next Fibonacci Number one at a time
   while p < max:
       yield p
       p, q = q, p + q


# Ask the user to enter the maximum number
n = int(input("Enter the number up to which you wish the Fibonacci series to be printed: \n"))

# Create a generator object
x = fib(n)
# Iterating over the generator object using for
# in a loop.
print("Resultant Series up to", n, "is :")
for i in x:
   print(i)

Input

Enter the number up to which you wish the Fibonacci series to be printed: 
5

Output

Resultant Series up to 5 is :
0
1
1
2
3

Explanation: yield in Python can be used as the return statement in a function precisely the way we have used in the above program. The function returns a generator that can be iterated upon instead of returning the output when done so.

Python iterates over the code until it reaches a yield line within the function. The function then transmits the produced value and pauses in that state without leaving. When the function is called again, its last paused state is remembered, and execution is begun from there. This will continue until the generator is exhausted.

Why use generators when the return statement is already present?

The most well-known feature of generators is their excellent memory efficiency. A standard function that returns a sequence will first construct the whole sequence in memory before returning the result. If the number of elements in the sequence is enormous, this is overkill. The generator implementation of such sequences, on the other hand, is memory-friendly and preferable since it only generates one item at a time.

Comparison of Iterators and Generators in Python

Now, let’s look at some distinctions between Iterators and Generators in Python:

Iterators	Generators
Iterators are the objects that use the next() method to get the next value of the sequence.	A generator is a function that produces or yields a sequence of values using a yield statement.
Classes are used to Implement the iterators.	Functions are used to implement the generator.
Every iterator is not a generator.	Every generator is an iterator.
Complex implementation of iterator protocols .i.e., iter() and next().	Generators in Python are simpler to code than do the custom iterator using the yield statement.
Iterators in python are less memory efficient.	Generators in Python are more memory efficient.
No local variables are used in Iterators.	All the local variables are stored before the yield statement.

Frequently Asked Questions

Key Takeaways

To summarise the session, we explored Iterators and Generators in python.

 A generator uses the ‘yield’ keyword in Python. A Python iterator, on the other hand, does not. Every time ‘yield’ pauses the loop in Python, the Python generator stores the states of the local variables. An iterator does not need local variables. All it requires is an iterable object to iterate on.

Don’t sit idle. Continue to practice the listed Implementations on your own. Keep an eye out for the fantastic articles.

Don’t stop here Ninja, get yourself enrolled in our Top-notch Courses from the best faculties.

Happy Learning!