Multiprocessing in Python: Running Multiple Processes Concurrently
Multithreading is useful for I/O-bound tasks, but for CPU-bound tasks, multiprocessing can be more effective. By running multiple processes, you can utilize multiple CPU cores, achieving true parallelism. In this blog post, we will explore how to use Python’s multiprocessing module to manage multiple processes
What is Multiprocessing?
Multiprocessing allows a program to run multiple processes concurrently, with each process having its own memory space. This is particularly beneficial for CPU-bound tasks that can fully utilize multiple CPU cores.
Why Use Multiprocessing?
- True Parallelism: Utilize multiple CPU cores for concurrent execution.
- Improved Performance: Handle CPU-bound tasks more efficiently.
- Independent Memory: Each process runs in its own memory space, reducing the risk of memory conflicts.
Example Code: Multiprocessing in Python
Let’s walk through an example where we calculate the squares of several numbers using multiprocessing.
Step 1: Import Required Modules
First, we need to import the multiprocessing and time modules.
from multiprocessing import Process
import time
Step 2: Define a Long-Running Function
We’ll create a function longSquare that simulates a time-consuming task by sleeping for a second before printing the square of a number.
def longSquare(n):
time.sleep(1)
print(f"Square of {n} is {n * n}")
Step 3: Create and Start Processes
We’ll create two processes to run the longSquare function concurrently.
# Create processes
p1 = Process(target=longSquare, args=(2,))
p2 = Process(target=longSquare, args=(3,))
# Start processes
p1.start()
p2.start()
# Wait for processes to complete
p1.join()
p2.join()
In this example, longSquare prints the square of the number directly. Since processes do not share memory, each process operates independently.
Step 4: Scaling Up with Multiple Processes
To handle a larger number of tasks, we can create a list of processes.
processes = []
# Create and start multiple processes
for n in range(10):
process = Process(target=longSquare, args=(n,))
processes.append(process)
process.start()
# Wait for all processes to complete
for process in processes:
process.join()
This code snippet creates and starts 10 processes, each calculating the square of a number from 0 to 9. Using a loop to start and join processes simplifies the process, especially when dealing with many processes.
Running with 100 Processes
For fun, let’s scale up to 100 processes and observe the performance improvement.
processes = []
# Create and start 100 processes
for n in range(100):
process = Process(target=longSquare, args=(n,))
processes.append(process)
process.start()
# Wait for all processes to complete
for process in processes:
process.join()
You will notice that the task completes much faster compared to running the calculations sequentially, as each process runs independently on different CPU cores.
Conclusion
Multiprocessing is a powerful tool for handling CPU-bound tasks in Python. By running multiple processes concurrently, you can achieve true parallelism and fully utilize your CPU cores. In this example, we demonstrated how to use the multiprocessing module to perform long-running computations in parallel. Experiment with multiprocessing in your projects to see the benefits firsthand.