Beyond the For-Loop: Higher Order Functions and List Comprehensions in Python
A higher order function is a fuction that operates on other functions, either by taking a function as its argument, or by returning a function. Since functions are first-class objects this is easy to do in Python, and the standard library comes with several higher order functions that make list operations more convenient. Let’s take a look at some of those.
Let’s say you have a list of integers and you only want the numbers that are even.
We could solve this with a for-loop:
integers = range(0, 10) even =  for i in integers: if i % 2 == 0: even.append(i) >>> even >>> [0, 2, 4, 6, 8]
Here we can use the filter function to make this more succinct. It takes as its arguments a function and an iterable and constructs a new iterable by applying the function to every element in the list.
even = filter(lambda x: x % 2 == 0, integers)
This solution is one line, whereas the for-loop was four lines. Note that we use a lambda function, an anynomous function, that can be convenient for short ad-hoc functions. We could also have written:
def is_even(x): return x % 2 == 0 even = filter(is_even, integers)
Finally, let’s see how we could have written this as a list comprehension:
even = [x for x in integers if x % 2 == 0]
What’s going on here? We simply say that we want each integer in integers if it is divisible by 2. It is in essence a flattened, more succint for loop. We don’t have to call any functions.
So now we have three solutions to the same problem. How do you know which one to use? It comes down to a matter of taste, speed and readability.
The for-loop is the longest, but it’s also easier to read and understand for someone new to Python or higher order functions. For someone who knows Python well the other versions might be both easier to read and write since they are shorter.
The higher order function and the list comprehension are both one-liners. Seasoned Python programmers tend to prefer list comprehensions. They state their intent clearly and can be read from left to right.
In this case the list comprehnsion solution is also faster, since it doesn’t make any function calls. For people coming from languages without list comprehensions (which would be most languages), the functional version would be more familiar.
Map applies a function to every element in a list. Unlike filter, it leaves the number of elements in the list unchanged. Let’s say we want to square every element in a list:
map(lambda x: x * x, integers) >>> [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
This could equally be written as a list comprehension:
square_list = [x * x for x in integers]
Again, the list comprehension is faster, and I would argue it is also more convenient and easier to read once the notation is familiar.
So far we’ve looked at functions that both consume and return lists. There are also numerous functions that consume a list but returns a single value.
Reduce works by applying a function that takes two arguments to the elements in the list, from left to right. Here we use it to get the product of all the integers in the list:
integers = range(1, 10) reduce(lambda x, y: x * y, integers) >>> 362880
Since list comprehnsions return lists, they are not applicable to these kind of problems.
The next three functions could all have been implemented by reduce(), but are convenient to have as stand-alone functions. Since they neither consume or return functions, they are not higher order functions.
Sum simply adds all the elements in a list:
sum(integers) >>> 45
You could do the same with reduce():
reduce(lambda x, y: x + y, integers)
Any and All
Any and all works just like the names suggest:
any([False, True, False]) >>> True all([False, True, False]) >>> False
The and() and any() functions are good examples of how functions that take an iterator can be combined with list comprehensions. To check for even numbers in a list:
any(x % 2 == 0 for x in integers) >>> True all(x % 2 == 0 for x in integers) >>> False
Where to go from here
Check out Python’s built in functions for other higher order functions. Also, take a look at Python’s itertools library.