Python is often criticized for its execution speed compared to languages like C++ or Rust. However, I’ve always believed that the bottleneck isn't usually the language itself—it’s how we use it. If you’ve ever felt your script crawling during a heavy data processing task, you aren't alone.
Over the past few months, I’ve been experimenting with various techniques to squeeze more performance out of my Python code. Here is what I’ve found to be the most effective strategies for leveling up your execution speed.
1.Leverage Built-in Functions and Libraries
The golden rule of Python optimization is: if it’s built-in, use it. Built-in functions are implemented in highly optimized C.
For instance, if you are performing mathematical operations or list manipulations, avoid writing manual for loops if a map() function or a list comprehension can do the job.
# Instead of this:
result = []
for i in range(1000000):
result.append(i * 2)
# Use list comprehension (Significantly faster):
result = [i * 2 for i in range(1000000)]
2.Vectorization with NumPy
When dealing with numerical data, standard Python lists are memory-heavy and slow. NumPy is your best friend. By using vectorized operations, you allow NumPy to perform calculations in pre-compiled C code, completely bypassing the Python interpreter's overhead.
import numpy as np
# Traditional loop
data = list(range(1000000))
def square_loop(arr):
return [x**2 for x in arr]
# Vectorized approach
arr = np.arange(1000000)
def square_numpy(arr):
return np.square(arr)
The performance difference here is often orders of magnitude.
3.Minimize Global Variable Access
In Python, looking up a global variable is slower than looking up a local one because of the way Python’s scope resolution works. If you have a loop that runs millions of times, copying a global variable into a local variable before entering the loop can provide a surprising speed boost.
import math
def compute_math(data):
# Bad: accessing math.sqrt in every iteration
# result = [math.sqrt(x) for x in data]
# Good: Localize the function
sqrt = math.sqrt
return [sqrt(x) for x in data]
4.Use Generators for Memory Efficiency
If you are processing massive datasets, you don't always need to store everything in memory at once. Generators allow you to process items one by one, keeping your memory footprint low and preventing system thrashing.
# Memory heavy
def get_large_list():
return [i for i in range(10000000)]
# Memory efficient
def get_large_generator():
for i in range(10000000):
yield i
5.When All Else Fails: Just-In-Time Compilation (PyPy/Numba)
If you have a CPU-bound function that simply must be fast, don't rewrite it in C++ immediately. Try Numba. It’s a JIT compiler that translates your Python functions into machine code at runtime.
from numba import jit
@jit(nopython=True)
def heavy_computation(n):
res = 0
for i in range(n):
res += i
return res
Final Thoughts
Python doesn't have to be slow. Often, it’s about choosing the right tool for the job—whether that’s leveraging C-backed libraries, optimizing your loops, or simply being smarter about how you manage memory.
postscript
6.Bonus: Choose the Right Data Structure (Lists vs. Sets)
Sometimes, rewriting your loops isn't enough; you need to look at what you are searching through.
If you frequently check whether an item exists in a collection using the in keyword, switching from a list to a set can give you an instant, massive speed boost. Checking item existence in a list takes linear time ($O(N)$), while a set takes constant time ($O(1)$).
# Slow: Searching in a list
large_list = list(range(10000000))
def search_in_list(target):
return target in large_list
# Blazing Fast: Searching in a set
large_set = set(range(10000000))
def search_in_set(target):
return target in large_set
For large datasets, the set approach is thousands of times faster because it uses a hash table under the hood. It’s a simple change, but it makes a world of difference.