These pages are no longer maintained. Please use current verison.

Computational Statistics in Python

Notebooks for each topic are in the GitHub repository

Topics

• Introduction to Python
  • Resources
  • Overview
  • Types
  • Operators
  • Names, assignment and identity
  • Naming conventions
  • Collections
  • Sets
  • Dictionary
  • Control Structures
  • Functions
  • Version Information
• Functions
  • Waht’s wrong with this code?
  • Refactoring to use functions
  • Function is re-usable
  • Fnctions can be passed in as arguments
  • Funcitons can also be returned by functions
  • Decorators
  • Functools
  • Where does Python search for modules?
  • Creating your own module
• Strings
  • Get “Through the Looking Glass”
  • Slice to get Jabberwocky
  • Find palindromic words in poem if any
  • Top 10 most frequent words
  • Words that appear exactly twice.
  • Trigrams
  • Find words in poem that are over-represented
  • Encode and decode poem using a Caesar cipher
  • Using Regular Expressions
  • Natural language processing
  • String formatting
• I/O
  • Text files
  • Loading a text file
  • Saving a text file
  • Web resources
  • Reading an unformatted web page
  • Getting a table from a URL
  • Flatten the nested dictionaries recursively
• Classes
  • Defining a new class
  • Docstrings
  • Making instance of a class
  • Class inheritance
• Using numpy
  • Resources
  • Array creation
  • Array manipulaiton
  • Array indexing
  • Calculations and broadcasting
  • Combining and splitting arrays
  • Reductions
  • Example: Calculating pairwise distance matrix using broadcasting and vectorization
  • Example: Consructing leave-one-out arrays
  • Generalized ufucns
  • Saving and loading NDArrays
  • Version information
• Graphics in Python
  • Resources
  • Matplotlib
  • Types of plots
  • Colors
  • Plot layouts
  • Seaborn
  • Using ggplot as an alternative to seaborn.
  • Version Information
• Data
  • Resources
  • Working with Series
  • DataFrame
  • Data conversions
  • Version information
• SQL
  • SQL via pandas DataFrames
  • Joins
  • User defined functions
• Machine Learning with sklearn
  • Resources
  • Example
  • Exploratory data analysis
  • Preprocessing
  • Dimension reduction
  • Classification
  • Using a Pipeline
• Linear Algebra Review
• Linear Algebra and Linear Systems
  • Motivation - Simultaneous Equations
  • Vector Spaces
  • Matrices and Linear Transformations
  • What does all this have to do with linear systems?
  • More Properties of Vectors, Vector Spaces and Matrices
  • Inner Products
  • Exercises
• Matrix Decompositions
  • LU Decomposition and Gaussian Elimination
  • Cholesky Decomposition
  • Matrix Decompositions for PCA and Least Squares
  • Eigendecomposition
  • QR decompositon
  • Singular Value Decomposition
  • Stabilty and Condition Number
  • Exercises
• Linear Algebra Examples
  • Resources
  • Exact solution of linear system of equations
  • Basic information about a matrix
  • Least-squares solution
  • Matrix Decompositinos
• Applications of Linear Alebra: PCA
  • Variance and covariance
  • Eigendecomposition of the covariance matrix
  • Covariance matrix as a linear transformation
  • PCA
  • Eigendecomposition of the covariance matrix
  • Change of basis via PCA
  • Graphical illustration of change of basis
  • Dimension reduction via PCA
• Symbolic Algebra with sympy
  • Basics
  • Simplify
  • Calculus
  • Working wiht matrices
  • Solving Algebraic and Differential Equations
  • Numerics
  • Statistics
• Optimization and Root Finding
  • Example: Maximum Likelihood Estimation (MLE)
  • Optimization in one dimension usually means finding roots of the derivative.
  • Main Issues in Root Finding in One Dimension
  • Bisection Method
  • Secant Method
  • Newton-Rhapson Method
  • Gauss-Newton
  • Inverse Quadratic Interpolation
  • Brent’s Method
• Algorithms for Optimization and Root Finding for Multivariate Problems
  • Optimization/Roots in n Dimensions - First Some Calculus
  • Convexity
  • Line Search Methods
  • Steepest Descent
  • Newton’s Method
  • Coordinate Descent
  • Solvers
  • GLM Estimation and IRLS
• Using optimization routines from scipy and statsmodels
  • Finding roots
  • Optimization Primer
  • Using scipy.optimize
  • Gradient descent
  • Optimization of standard statistical models
• Expecation Maximization
  • Outline
  • Jensen’s inequality
  • Maximum likelihood with complete information
  • Incomplete information
  • Mixture Models
  • Gaussian mixture models
  • Using EM
  • Vectorized version
  • Vectorization with Einstein summation notation
  • Comparison of EM routines
• Random Numbers
  • Pseudorandom number generators (PRNG)
  • From standard uniform to other distributions
  • Using the numpy.random and scipy.stats PRNGs
• Resampling and Monte Carlo Simulations
  • Setting the random seed
  • Adjusting p-values for multiple testing
  • Jackknife estimate of parameters
  • Leave one out cross validation (LOOCV)
  • Check with R
  • Estimating the CDF
  • Estimating the PDF
• Numerical Evaluation of Integrals
  • Quadrature
  • Monte Carlo integration
  • Variance Reduction
  • Quasi-random numbers
• Metropolis and Gibbs Sampling
  • Island hopping
  • Bayesian Data Analysis
  • Hierarchical models
• Using Auxiliary Variables in MCMC proposals
  • Slice sampling
  • A simple slice sampler example
  • Hamiltonian Monte Carlo (HMC)
  • Hamiltonian systems
  • Finite difference methods
  • From Hamiltonians to probability distributions
• Using PyMC3
  • Coin bias
  • Univariate normal distribution
  • Linear regression
  • Using the GLM module
  • Robust linear regression
  • Logistic regression
  • Logistic regression for classification
  • Estimating parameters of a logistic model
  • Hierarchical model
  • Multivariate normal
  • Gaussian Mixture Model
  • Changepoint detection
  • Bandits
• PyStan
  • Useful links
  • Coin toss
  • Estimating mean and standard deviation of normal distribution
  • Estimating parameters of a linear regression model
  • Simple Logistic model
  • Estimating parameters of a logistic model
  • Using a hierarchical model
• Introduction to C
  • Compiled vs Interpreted Languages
  • Important Definitions
  • A tutorial example - coding a Fibonacci function in C
  • C Basics
  • Using make to compile C programs
  • Exercise
  • Obfuscated C (how not to write C code)
• Introduction to C++
  • Hello, world
  • Namespaces
  • Looping
  • Functions and Lambdas
  • Templates
  • Iterators
  • Containers
  • Algorithms
  • Function pointers
  • Random numbers
  • Numeric library
  • Collected source code
• Review and Quiz on C and C++
  • What you should know about C
  • What you should know about C++
• Code Optimization
  • Testing code
  • Timing and profiling code
  • Data structures and algorithms
• Foreign Function Interface
  • Wrapping functions written in C
  • C++
  • Wrap an R function from libRmath using ctypes
  • Fortran
  • Interfacing with R
• Just-in-time compilation (JIT)
  • Using numexpr
  • Using numba
  • Using numba vectorize and guvectoize
  • Parallelization with vectorize and guvectorize
• Cython
  • Resources
  • Using Cython annnotations to identify bottlenecks
  • Using Cython cdefs and directives
  • Parallel execution with Cython
• Making Python faster
  • Fib
  • Parse_int
  • Quicksort
  • Mandel
  • Pi_sum
  • Rand_mat_stat
  • Rand_mat_mul
  • Comparison
• Comparsion of optimizaiton approaches
  • Interfacing with compiled languages
  • From Python \(\rightarrow\) Compiled langauges
  • From Compiled langauges \(\rightarrow\) Python
  • Bake-off
  • Summary
• Using pybind11
  • A first example of using pybind11
  • Using cppimport
  • Vectorizing functions for use with numpy arrays
  • Using numpy arrays as function arguments and return values
  • Using the C++ eigen library to calculate matrix inverse and determinant
• Parallel Programming
  • Embarrassingly parallel programs
  • Executing parallel code
  • Coming highlights
• Multi-Core Parallelism
  • Vanilla Python
  • Using numba to speed up computation
  • Using cython to speed up computation
  • The concurrent.futures module
  • Using processes in parallel with ProcessPoolExecutor
  • Using processes in parallel with ThreadPoolExecutor
  • Turning off the GIL in cython
  • Using processes in parallel with ThreadPoolExecutor and nogil
  • Using multiprocessing
  • Common issues with use of shared memory in parallel programs
• Using ipyparallel
  • Starting engines
  • Basic concepts of ipyparallel
  • Working with compiled code
  • Using parallel magic commands
• Biggish Data
  • 1 billion numbers
  • Using Dask
  • Using Blaze
• Efficient storage of data in memeory
  • Selective retrieval from disk-based storage
  • Storing numbers
  • Storing strings
  • Data Sketches
• Introduction to Spark
  • Learning objectives
  • Installation
  • Resources
  • Overview of Spark
  • Languages supported
  • Architecture of a Spark Application
  • SparkContext
  • Actions and transforms
  • Working with key-value pairs
  • Persisting data
• Using Spark Efficiently
  • Resources
  • Accumulators
  • Broadcast Variables
  • The Spark Shuffle and Partitioning
  • Piping to External Programs
• Spark SQL
  • Resources
  • DataFrame from pandas
  • DataFrame from CSV files
  • DataFrame from JSON files
  • DataFrame from SQLite3
  • DataSets
• Spark MLLib
  • MLLib Pipeline
  • Unsupervised Learning
  • Supervised Learning
  • Using the newer ml pipeline
  • Spark MLLIb and sklearn integration
• Spark on Cloud
  • Azure
  • AWS
  • Know your AWS public and private access keys
  • Know your AWS EC2 key-pair
  • Install AWS command line client
  • Configure the AWS command line client
  • Create a cluster
  • Get information about the cluster
  • Connect to the cluster via ssh
  • Note the IP address that is returned
  • Run pyspark
  • Run the Zepellin notebook
  • Connect to Zeppelin notebook
  • Create notebook and run Spark within it
  • Terminate the cluster
• Further Reading

Setup

• Setting up a local install of Jupyter
• Customizing the Jupyter notebook

Homework Solutions

Notebooks containing homework problem sets can be found in the GitHub repository

• Homework 01
• Homework 02
• Homework 03
• Homework 04
• Homework 05
• Homework 06
• Homework 07
• Homework 08
• Homework 09
• Homework 10

Indices and tables

• Index
• Module Index
• Search Page