Multiprocessing in Python

Most of the time we run multiple applications on our devices. The devices we use in this era are capable of handling these multiple tasks done simultaneously. This is because of the concept of multiprocessing.

In this article, we will learn multiprocessing and doing this in Python using the module multiprocessing. We will also learn different methods and classes in this module.

Introduction to Multiprocessing

Previously, the systems had single processors. When we had to do multiple tasks, the processor handled them by allocating itself for one task at a time and switching between each of the tasks. This can be thought of as a single owner handling all the customers in a shop.

With the increase in the need for more tasks, the complexity and the load on the processor increased. And the efficiency and the speed decreased. This leads to multi-processor systems.
This is the situation where the owner hired people to handle the customers.

Most of the devices we have now contained a multiprocessing system. These systems may have any of the below architecture:

1. A multiprocessor- a device with more than one central processor.
2. A multi-core processor- a single component for computation with more than one independent processing unit/ cores.

Multiprocessing is the ability of the system to handle multiple processes simultaneously and independently. In a multiprocessing system, the applications are broken into smaller routines and the OS gives threads to these processes for better performance.

Multiprocessing in Python

Python provides a multiprocessing module that includes an API, similar to the threading module, to divide the program into multiple processes. Let us see an example,

Example of multiprocessing in Python:

import multiprocessing #importing the module
 
def even(n): #function to print all even numbers till n
    for i in range(0,n,2):
        print("even:",i)

def odd(n): #function to print all odd numbers till n
    for i in range(1,n,2):
        print("odd:",i)

if __name__=="__main__":
    # creating processes for each of the functions
    prc1 = multiprocessing.Process(target=even, args=(15, ))
    prc2 = multiprocessing.Process(target=odd, args=(15, ))

    # starting the 1st process
    prc1.start()
    # starting the 2nd process
    prc2.start()

    # waiting until 1st process is finished
    prc1.join()
    # waiting until 2nd process is finished
    prc2.join()

    # both processes finished
    print("END!")

Output:

In this example,

1. We imported the multiprocessor module

2. Then created two functions. One function prints even numbers and the other prints odd numbers less than the given value of ‘n’.

3. We created the instances of these functions in the Process class. We did this by giving the functions as the target argument. We pass the corresponding inputs to the targets as the args argument.

4. Then we executed the functions using the start() function in the Process class.

5. We used the join() function for the process to wait till the previous one finishes.

6. Then we wrote the print statement that displays ‘END’, which gets executed once the processes are completed.

Getting information about the processes in Python

We can get the information about the processes running like id and name. We can also check if the process is currently alive or not.

First, let us see how to get id and check if the process is alive or not.

Getting id of the processes and checking if it is alive

We can use the getpid() function ins the os module to get the id of the processes. And to know if the process is alive we can use the is_alive() function. The below example shows the state of the process in different parts of the program.

Example of getting id and checking if the process is alive:

import multiprocessing
from multiprocessing import Process
import os

def func1(): #function to print all even numbers till n
    print("Id of Funct1: ",os.getpid())

def func2(): #function to print all odd numbers till n
    print("Id of Funct2: ",os.getpid())

if __name__=="__main__":
    print("Id of the main process: ",os.getpid())
    # creating processes for each of the functions
    prc1 = multiprocessing.Process(target=func1)
    prc2 = multiprocessing.Process(target=func2)

    # starting the 1st process
    prc1.start()
    # starting the 2nd process
    prc2.start()
   
    print('When the process 1 and 2 started')
    alive1='Yes' if prc1.is_alive() else 'No'
    print("Is process 1 alive?",alive1)
    alive2='Yes' if prc2.is_alive() else 'No'
    print("Is process 2 alive?",alive2)

    # waiting until 1st process is finished
    prc1.join()
    print('When process 1 is completed and 2 is continuing:')
    alive1='Yes' if prc1.is_alive() else 'No'
    print("Is process 1 alive?",alive1)
    alive2='Yes' if prc2.is_alive() else 'No'
    print("Is process 2 alive?",alive2)
   
    # waiting until 2nd process is finished
    prc2.join()
    print('When both the processes are completed:')
    alive1='Yes' if prc1.is_alive() else 'No'
    print("Is process 1 alive?",alive1)
    alive2='Yes' if prc2.is_alive() else 'No'
    print("Is process 2 alive?",alive2)

    # both processes finished
    print("END!")

Output:

From the ids, we can see that the two processes are different from the main process.

Getting the name of the process

We can use the name() function multiprocessing module. For example,

Example of getting the name of the processes:

import multiprocessing
from multiprocessing import Process,current_process
import os

def func1(): #function to print all even numbers till n
    print("Name of Process 1: ",current_process().name)

def func2(): #function to print all odd numbers till n
    print("Name of Process 2: ",current_process().name)

if __name__=="__main__":
    
    # creating processes for each of the functions
    prc1 = multiprocessing.Process(target=func1)
    prc2 = multiprocessing.Process(target=func2)

    # starting the 1st process
    prc1.start()
    # starting the 2nd process
    prc2.start()
   
    # waiting until 1st process is finished
    prc1.join()  
    # waiting until 2nd process is finished
    prc2.join()
    
    # both processes finished
    print("END!")

Output:

We can see that we are not getting any name or the function of the process, but we are just getting the names Process-1 and Process-2. We have an option to set the name by giving another keyword argument called ‘name’ while creating the instance of the Process class. For example,

Example of getting the name of the processes:

import multiprocessing
from multiprocessing import Process,current_process
import os

def func1(): #function to print all even numbers till n
    print("Name of Process 1: ",current_process().name)

def func2(): #function to print all odd numbers till n
    print("Name of Process 2: ",current_process().name)

if __name__=="__main__":
    
    # creating processes for each of the functions
    prc1 = multiprocessing.Process(target=func1,name='Funct 1')
    prc2 = multiprocessing.Process(target=func2,name='Funct 2')

    # starting the 1st process
    prc1.start()
    # starting the 2nd process
    prc2.start()
   
    # waiting until 1st process is finished
    prc1.join()  
    # waiting until 2nd process is finished
    prc2.join()
    
    # both processes finished
    print("END!")

Output:

Now we can see that we got the names we set.

Locks in Multiprocessing in Python

Similar to multithreading, multiprocessing in Python also supports locks. We can set the lock to prevent the interference of threads. When the lock is set, a process starts only when the previous process is finished and the lock is released.

We can do this by importing the Lock object from the multiprocessing module. Here also, there are two functions:

1. acquire(blocking=True, timeout=-1):

This function acquires a lock, which can be either blocking or non-blocking.

2. release():

To function releases the lock.

Example of getting the name of the processes:

from multiprocessing import Process, Lock
lck=Lock()
a=0
def func_print(item):
    lck.acquire()
    try:
        for i in range(5):
            print(item)
    finally:
        lck.release()
if __name__=="__main__":
    items=['Hi','Hello','Bye']
    for item in items:
        p=Process(target=func_print,args=(item,))
        p.start()

Output:

We can see that the processes are executing only when the previous one is finished.

Pool Class in Python Multiprocessing

The pool is a class in the multiprocessing module that distributes the tasks to the available processors in FIFO (First In First Out) manner.

Example of Pool class:

from multiprocessing import Pool
def square(n):
    return n**2
if __name__=='__main__':
    numbers=[1,5,9]
    pool=Pool(processes=3)
    print(pool.map(square,numbers))   

Output:

In the example, we are creating an instance of the Pool() class. The map() function takes the function and the arguments as iterable. Then it runs the function for every element in the iterable.

Let us see another example, where we use another function of Pool() class. This is map_async() function that assigns the job to the worker pool.

Example of Pool class:

from multiprocessing import Pool
def square(n):
    return n**2
if __name__=='__main__':
    pool=Pool(processes=3)
    res=pool.apply_async(square,(10,))
    print(res.get()) 

Output:

Here, we are using the get() method to get the return value of the function.

Conclusion

We learned about multiprocessing and implementing it in Python. We also learned to get information about processes, using Locks and the pool.

Hope you enjoyed reading this article. Happy learning!