Python review of concepts

Mainly to point out useful aspects of Python you may have glossed over. Assumes you already know Python fairly well.

Python as a language

Why Python?

• Huge community - especially in data science and ML

• Easy to learn

• Batteries included

• Extensive 3rd party libraries

• Widely used in both industry and academia

• Most important “glue” language bridging multiple communities

import __hello__

Hello world!

Versions

• Only use Python 3 (current release version is 3.8, container is 3.7)

• Do not use Python 2

import sys

sys.version

'3.8.5 (default, Jul 21 2020, 10:48:26) \n[Clang 11.0.3 (clang-1103.0.32.62)]'

Multi-paradigm

Procedural

x = []
for i in range(5):
    x.append(i*i)
x

[0, 1, 4, 9, 16]

Functional

list(map(lambda x: x*x, range(5)))

[0, 1, 4, 9, 16]

Object-oriented

class Robot:
    def __init__(self, name, function):
        self.name = name
        self.function = function

    def greet(self):
        return f"I am {self.name}, a {self.function} robot!"

fido = Robot('roomba', 'vacuum cleaner')

fido.name

'roomba'

fido.function

'vacuum cleaner'

fido.greet()

'I am roomba, a vacuum cleaner robot!'

Dynamic typing

Complexity of a + b

1 + 2.3

3.3

type(1), type(2.3)

(int, float)

'hello' + ' world'

'hello world'

[1,2,3] + [4,5,6]

[1, 2, 3, 4, 5, 6]

import numpy as np

np.arange(3) + 10

array([10, 11, 12])

Several Python implementations!

• CPtyhon

• Pypy

• IronPython

• Jython

Global interpreter lock (GIL)

• Only applies to CPython

• Threads vs processes

• Avoid threads in general

• Performance not predictable

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def f(n):
    x = np.random.uniform(0,1,n)
    y = np.random.uniform(0,1,n)
    count = 0
    for i in range(n):
        if x[i]**2 + y[i]**2 < 1:
            count += 1
    return count*4/n

n = 100000
niter = 4

%%time

[f(n) for i in range(niter)]

CPU times: user 525 ms, sys: 3.21 ms, total: 528 ms
Wall time: 528 ms

[3.1392, 3.153, 3.14876, 3.14132]

%%time

with ThreadPoolExecutor(4) as pool:
    xs = list(pool.map(f, [n]*niter))
xs

CPU times: user 549 ms, sys: 6.37 ms, total: 556 ms
Wall time: 546 ms

[3.14536, 3.1468, 3.13868, 3.14756]

%%time

with ProcessPoolExecutor(4) as pool:
    xs = list(pool.map(f, [n]*niter))
xs

---------------------------------------------------------------------------
BrokenProcessPool                         Traceback (most recent call last)
<timed exec> in <module>

/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/process.py in _chain_from_iterable_of_lists(iterable)
    482     careful not to keep references to yielded objects.
    483     """
--> 484     for element in iterable:
    485         element.reverse()
    486         while element:

/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py in result_iterator()
    609                     # Careful not to keep a reference to the popped future
    610                     if timeout is None:
--> 611                         yield fs.pop().result()
    612                     else:
    613                         yield fs.pop().result(end_time - time.monotonic())

/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Coding in Python

import this

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Coding conventions

• PEP 8

• Avoid magic numbers

• Avoid copy and paste

• extract common functionality into functions

Style Guide for Python Code

Data types

• Integers

  • Arbitrary precision

  • Integer division operator

  • Base conversion

  • Check if integer

import math

n = math.factorial(100)

n

93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000

f'{n:,}'

'93,326,215,443,944,152,681,699,238,856,266,700,490,715,968,264,381,621,468,592,963,895,217,599,993,229,915,608,941,463,976,156,518,286,253,697,920,827,223,758,251,185,210,916,864,000,000,000,000,000,000,000,000'

h = math.sqrt(3**2 + 4**2)

h

5.0

h.is_integer()

True

• Floats

  • Checking for equality

  • Catastrophic cancellation

• Complex

x = np.arange(9).reshape(3,3)
x = x / x.sum(axis=0)
λ = np.linalg.eigvals(x)

λ[0]

0.9999999999999993

λ[0] == 1

False

math.isclose(λ[0], 1)

True

def var(xs):
    """Returns variance of sample data."""

    n = 0
    s = 0
    ss = 0

    for x in xs:
        n +=1
        s += x
        ss += x*x

    v = (ss - (s*s)/n)/(n-1)
    return v

xs = np.random.normal(1e9, 1, int(1e6))

var(xs)

13287.56835956836

np.var(xs)

1.0003007438816822

• Boolean

  • What evaluates as False?

stuff = [[], [1], {},'', 'hello', 0, 1, 1==1, 1==2]
for s in stuff:
    if s:
        print(f'{s} evaluates as True')
    else:
        print(f'{s} evaluates as False')

[] evaluates as False
[1] evaluates as True
{} evaluates as False
 evaluates as False
hello evaluates as True
0 evaluates as False
1 evaluates as True
True evaluates as True
False evaluates as False

• String

  • Unicode by default

  • b, r, f strings

u'\u732b'

'猫'

String formatting

• Learn to use the f-string.

import string

char = 'e'
pos = string.ascii_lowercase.index(char) + 1
f"The letter {char} has position {pos} in the alphabet"

'The letter e has position 5 in the alphabet'

n = int(1e9)
f"{n:,}"

'1,000,000,000'

x = math.pi

f"{x:8.2f}"

'    3.14'

import datetime
now = datetime.datetime.now()
now

datetime.datetime(2020, 11, 11, 19, 23, 45, 578067)

f"{now:%Y-%m-%d %H:%M}"

'2020-11-11 19:23'

Data structures

• Immutable - string, tulle

• Mutable - list, set, dictionary

• Collections module

• heapq

import collections

[x for x in dir(collections) if not x.startswith('_')]

['ChainMap',
 'Counter',
 'OrderedDict',
 'UserDict',
 'UserList',
 'UserString',
 'abc',
 'defaultdict',
 'deque',
 'namedtuple']

Functions

• *args, **kwargs

• Care with mutable default values

• First class objects

• Anonymous functions

• Decorators

def f(*args, **kwargs):
    print(f"args = {args}") # in Python 3.8, you can just write f'{args = }'
    print(f"kwargs = {kwargs}")

f(1,2,3,a=4,b=5,c=6)

args = (1, 2, 3)
kwargs = {'a': 4, 'b': 5, 'c': 6}

def g(a, xs=[]):
    xs.append(a)
    return xs

g(1)

[1]

g(2)

[1, 2]

h = lambda x, y, z: x**2 + y**2 + z**2

h(1,2,3)

14

from functools import lru_cache

def fib(n):
    print(n, end=', ')
    if n <= 1:
        return n
    else:
        return fib(n-2) + fib(n-1)

fib(10)

10, 8, 6, 4, 2, 0, 1, 3, 1, 2, 0, 1, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1, 7, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1, 6, 4, 2, 0, 1, 3, 1, 2, 0, 1, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1, 9, 7, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1, 6, 4, 2, 0, 1, 3, 1, 2, 0, 1, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1, 8, 6, 4, 2, 0, 1, 3, 1, 2, 0, 1, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1, 7, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1, 6, 4, 2, 0, 1, 3, 1, 2, 0, 1, 5, 3, 1, 2, 0, 1, 4, 2, 0, 1, 3, 1, 2, 0, 1,

55

@lru_cache(maxsize=100)
def fib_cache(n):
    print(n, end=', ')
    if n <= 1:
        return n
    else:
        return fib_cache(n-2) + fib_cache(n-1)

fib_cache(10)

10, 8, 6, 4, 2, 0, 1, 3, 5, 7, 9,

55

Classes

• Key idea is encapsulation into objects

• Everything in Python is an object

• Attributes and methods

• What is self?

• Special methods - double underscore methods

• Avoid complex inheritance schemes - prefer composition

• Learn “design patterns” if interested in OOP

(3.0).is_integer()

True

'hello world'.title()

'Hello World'

class Student:
    def __init__(self, first, last):
        self.first = first
        self.last = last

    @property
    def name(self):
        return f'{self.first} {self.last}'

s = Student('Santa', 'Claus')

s.name

'Santa Claus'

Enums

Use enums readability when you have a discrete set of CONSTANTS.

from enum import Enum

class Day(Enum):
    MON = 1
    TUE = 2
    WED = 3
    THU = 4
    FRI = 5
    SAT = 6
    SUN = 7

for day in Day:
    print(day)

Day.MON
Day.TUE
Day.WED
Day.THU
Day.FRI
Day.SAT
Day.SUN

NamedTuple

from collections import namedtuple

Student = namedtuple('Student', ['name', 'email', 'age', 'gpa', 'species'])

abe = Student('Abraham Lincoln', 'abe.lincoln@gmail.com', 23, 3.4, 'Human')

abe.species

'Human'

abe[1:4]

('abe.lincoln@gmail.com', 23, 3.4)

Data Classes

Simplifies creation and use of classes for data records.

Note: NamedTuple serves a similar function but are immutable.

from dataclasses import dataclass

@dataclass
class Student:
    name: str
    email: str
    age: int
    gpa: float
    species: str = 'Human'

abe = Student('Abraham Lincoln', 'abe.lincoln@gmail.com', age=23, gpa=3.4)

abe

Student(name='Abraham Lincoln', email='abe.lincoln@gmail.com', age=23, gpa=3.4, species='Human')

abe.email

'abe.lincoln@gmail.com'

abe.species

'Human'

Note

The type annotations are informative only. Python does not enforce them.

Student(*'abcde')

Student(name='a', email='b', age='c', gpa='d', species='e')

Imports, modules and namespaces

• A namespace is basically just a dictionary

• LEGB

• Avoid polluting the global namespace

[x for x in dir(__builtin__) if x[0].islower()][:8]

['abs', 'all', 'any', 'ascii', 'bin', 'bool', 'breakpoint', 'bytearray']

x1 = 23

def f1(x2):
    print(locals())
    # x1 is global (G), x2 is enclosing (E), x3 is local
    def g(x3):
        print(locals())
        return x3 + x2 + x1
    return g

x = 23

def f2(x):
    print(locals())
    def g(x):
        print(locals())
        return x
    return g

g1 = f1(3)
g1(2)

{'x2': 3}
{'x3': 2, 'x2': 3}

28

g2 = f2(3)
g2(2)

{'x': 3}
{'x': 2}

2

Loops

• Prefer vectorization unless using numba

• Difference between continue and break

• Avoid infinite loops

• Comprehensions and generator expressions

import string

{char: ord(char) for char in string.ascii_lowercase}

{'a': 97,
 'b': 98,
 'c': 99,
 'd': 100,
 'e': 101,
 'f': 102,
 'g': 103,
 'h': 104,
 'i': 105,
 'j': 106,
 'k': 107,
 'l': 108,
 'm': 109,
 'n': 110,
 'o': 111,
 'p': 112,
 'q': 113,
 'r': 114,
 's': 115,
 't': 116,
 'u': 117,
 'v': 118,
 'w': 119,
 'x': 120,
 'y': 121,
 'z': 122}

Iterations and generators

• The iterator protocol

  • __iter__ and __next__

  • iter()

  • next()

• What happens in a for loop

• Generators with yield and yield from

class Iterator:
    """A silly class that implements the Iterator protocol and Strategy pattern.

    start = start of range to square
    stop = end of range to square
    """
    def __init__(self, start, stop, func):
        self.start = start
        self.stop = stop
        self.func = func

    def __iter__(self):
        self.n = self.start
        return self

    def __next__(self):
        if self.n >= self.stop:
            raise StopIteration
        else:
            x = self.func(self.n)
            self.n += 1
            return x

sq = Iterator(0, 5, lambda x: x*x)

list(sq)

[0, 1, 4, 9, 16]

Generators

Like functions, but lazy.

def cycle1(xs, n):
    """Cuycles through values in xs n times."""

    for i in range(n):
        for x in xs:
            yield x

list(cycle1([1,2,3], 4))

[1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]

for x in cycle1(['ann', 'bob', 'stop', 'charles'], 1000):
    if x == 'stop':
        break
    else:
        print(x)

ann
bob

def cycle2(xs, n):
    """Cuycles through values in xs n times."""

    for i in range(n):
        yield from xs

list(cycle2([1,2,3], 4))

[1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]

Because they are lazy, generators can be used for infinite streams.

def fib():
    a, b = 1, 1
    while True:
        yield a
        a, b = b, a + b

for n in fib():
    if n > 100:
        break
    print(n, end=', ')

1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89,

You can even slice infinite generators. More when we cover functional programming.

import itertools as it

list(it.islice(fib(), 5, 10))

[8, 13, 21, 34, 55]