Python: Functions

To a large extent, data analysis consists of a sequence of data transformations using functions. Given the centrality of functions in this course, the notebook goes into some depth into Python function construction and usage.

Built-in functions

See Python docs

xs = ['apple', 'pear', 'grape', 'orange', 'rambutan', 'durian', 'longan', 'mango']

sorted(xs)

['apple', 'durian', 'grape', 'longan', 'mango', 'orange', 'pear', 'rambutan']

sorted(xs, key=len)

['pear', 'apple', 'grape', 'mango', 'orange', 'durian', 'longan', 'rambutan']

sorted(xs, key=len, reverse=True)

['rambutan', 'orange', 'durian', 'longan', 'apple', 'grape', 'mango', 'pear']

max(xs)

'rambutan'

max(xs, key=len)

'rambutan'

min(xs)

'apple'

min(xs, key=len)

'pear'

sum(map(len, xs))

45

Custom functions

It is very simple to write a function. The docstring (a triple quoted string with 1 or more lines of function documentation is not mandatory but highly recommended).

def power(x, n=2):
    """Returns x to the nth power.

    n has a default value of 2."""

    return x**n

help(power)

Help on function power in module __main__:

power(x, n=2)
    Returns x to the nth power.

    n has a default value of 2.

Use default value for exponent

power(3)

9

Give explicit exponent value

power(3, 4)

81

Give named arguments out of order

power(n=0.5, x=3)

1.7320508075688772

With arbitrary arguments

def f(a, b, *args, **kwargs):
    """Example to illustrate use * and ** arguments."""
    return a, b, args, kwargs

a, b, args, kwargs = f(1, 2, 3, 4, 5, x=10, y=11, z = 13)

a, b

(1, 2)

args

(3, 4, 5)

kwargs

{'x': 10, 'y': 11, 'z': 13}

Required keyword arguments

def f1(a, b, *, c, d):
    """c and d MUST be given as keyword arguments."""
    return a, b, c, d

f1(1, 2, c=3, d=4)

(1, 2, 3, 4)

try:
    f1(1, 2, 3, 4)
except Exception as e:
    print(e)

f1() takes 2 positional arguments but 4 were given

def f2(a, b, *args, c, d, **kwargs):
    """Combining requyired and optional arguments."""
    return a, b, c, d, args, kwargs

a, b, c, d, args, kwargs = f2(1, 2, 3, 4, c=5, d=6, e=7, f=8)

a, b,

(1, 2)

c, d

(5, 6)

args

(3, 4)

kwargs

{'e': 7, 'f': 8}

try:
    f2(1, 2, 3, 4, 5, 6, e=7, f=8)
except Exception as e:
    print(e)

f2() missing 2 required keyword-only arguments: 'c' and 'd'

All arguments are keyword only

def f3(*, a, b, c):
    """a, b and c  must all be given as keyword arguments."""
    return a, b, c

f3(c=1, b=2, a=4)

(4, 2, 1)

try:
    f3(4, 2, 1)
except Exception as e:
    print(e)

f3() takes 0 positional arguments but 3 were given

Expanding function arguments

def f4(a, b, c, d):
    return a, b, c, d

a = 1
bc = [2, 3]
d = 4

f4(a, *bc, d)

(1, 2, 3, 4)

Function annotations

You can indicate the type of the arguments and return values using function annotations. Python itself does not do anything with these except put them in a dictionary under the __annotations__ attribute, but 3rd party packages may use them if present. You will not be expected to use function annotations in this course.

def power2(x : float, n : dict(type=float, help='exponent') =2) -> float:
    """Returns x to the nth power.

    n has a default value of 2."""

    return x**n

power2.__annotations__

{'n': {'help': 'exponent', 'type': float}, 'return': float, 'x': float}

Lambda functions

Also known as anonymous functions. These are often used to construct one-use-only short functions for higher order functions such as map, filter and reduce.

power2 = lambda x, n=2: x**n

power2(3)

9

power2(3, 4)

81

Recursive functions

A recursive function is one that calls itself. Python is not optimized for such functions and may crash if the recursion goes too deep. There is always a non-recursive version for any recursive algorithm that should be used instead, but this may not be obvious. Unlike functional languages with tail call optimization, recursive functions are rarely used in Python as they are usually slower and consume more memory than the equivalent non-recursive version.

Recursive functions consist of

• a base case which terminates the computation
• a recursive call that MUST eventually end up in the base case

def factorial_1(n):
    """Recursive factorial function."""

    if n == 0:
        return 1
    else:
        return n * factorial_1(n-1)

factorial_1(50)

30414093201713378043612608166064768844377641568960512000000000000

from functools import reduce

def factorial_2(n):
    """Non-recursive version."""
    return reduce(lambda a, b: a*b, range(1, n+1))

factorial_2(50)

30414093201713378043612608166064768844377641568960512000000000000

The non-recursive version is usually more time and memory efficient

%timeit factorial_1(50)

14 µs ± 89.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit factorial_2(50)

10.6 µs ± 71.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Higher order functions

A higher order function is a function that takes another function as an argument or returns a function. Since functions are “first class” in Python, they can be treated in the same way as any other value. In particular, they can be used as function arguments and/or as return values.

Classical higher order functions are map, filter and reduce, here presented with more Pythonic versions using built-ins and comprehensions. However, map, filter and reduce are still important because often parallel and distributed code is most efficiently coded using these functional operators.

Map

list(map(lambda x: x**3, range(5)))

[0, 1, 8, 27, 64]

[x**3 for x in range(5)]

[0, 1, 8, 27, 64]

Filter

list(filter(lambda x: x % 2 == 0,
           range(5)))

[0, 2, 4]

[x for x in range(5) if x % 2 == 0]

[0, 2, 4]

Reduce

from functools import reduce

reduce(lambda x, y: x+y,
       map(lambda x: x**3, range(5)))

100

sum([x**3 for x in range(5)])

100

Using the operator module

The operator modules provides function equivalents for all of Python’s operators that are convenient for use in higher order functions.

import operator as op

reduce(op.add, map(lambda x: x**3, range(5)))

100

Custom function taking function arguments

def f(a, b, g, h):
    """Function taking functions g and h as arguments."""
    return g(a) + h(b)

f(2, 2, lambda x: x**2, lambda x: x**3)

12

f('abc', -2, len, abs)

5

Functions returning functions

The partial function takes a function as argument and returns another function

from functools import partial

def f1(a, b, c):
    """A function with 3 arguments."""
    return a, b, c

f1(1, 2, 3)

(1, 2, 3)

f2 takes a single argument since b and c have been given values by partial

f2 = partial(f1, b=12, c=13)

f2(11)

(11, 12, 13)

Custom function returning a function

def timed(f, *args, **kwargs):
    """Decorates the function f with time takin in seconds."""
    import time

    def func(*args, **kwargs):
        start = time.time()
        result = f(*args, **kwargs)
        elapsed = time.time() - start
        return elapsed, result
    return func

def my_sum(xs):
    s = 0
    for x in xs:
        s += x
    return s

my_sum(range(10000000))

49999995000000

my_sum2 = timed(my_sum)
my_sum2(range(10000000))

(0.8492159843444824, 49999995000000)

Decorators

There is syntactic sugar to decorate functions when using functions such as the timed function above.

@timed
def my_sum3(xs):
    s = 0
    for x in xs:
        s += x
    return s

my_sum3(range(10000000))

(0.9857957363128662, 49999995000000)

Giving decorators their own arguments

You will come across packages that provide decorators which can take arguments. This is one way that they can be implemented. See if you can follow how it works!

def timed2(fudge_factor=0.0):
    """Decorator with decorartor arguments."""
    def timed(f):
        import time
        def func(*args, **kwargs):
            start = time.time()
            result = f(*args, **kwargs)
            elapsed = fudge_factor + time.time() - start
            return elapsed, result
        return func
    return timed

@timed2(fudge_factor=10)
def my_sum4(xs):
    s = 0
    for x in xs:
        s += x
    return s

my_sum4(range(10000000))

(11.009344100952148, 49999995000000)