Welcome to STA663-2020’s documentation!

Contents:

• Text
  • Objectives
  • Strings
  • The string module
  • String methods
  • Formatting strings
  • Encodings
  • Using regular expressions
  • Examples
• Numbers
  • Learning objectives
  • The ndarray: Vectors, matrices and tenosrs
  • Creating ndarrays
  • Indexing
  • Universal functions
  • Margins and the axis argument
  • Broadcasting
  • Masking
  • Combining ndarrays
  • Splitting ndarrays
  • Saving and loading arrays
  • Vectorization
• Introduction to pandas
  • Series and Data Frames
  • Creating Data Frames
  • Indexing Data Frames
  • Structure of a Data Frame
  • Selecting, Renaming and Removing Columns
  • Selecting, Renaming and Removing Rows
  • Transforming and Creating Columns
  • Sorting Data Frames
  • Summarizing
  • Split-Apply-Combine
  • Combining Data Frames
  • Fixing common DataFrame issues
  • Reshaping Data Frames
  • Chaining commands
• Graphics and Visualization in Python
  • Pandas
  • Matplotlib
  • Seaborn
  • plotnine
• Functional programming in Python (operator, functional, itertoools, toolz)
  • Pure functions
  • Recursive functions
  • Anonymous functions
  • Lazy evaluation
  • Higher order functions
  • Example: Flattening a nested list
  • Closure
  • Decorators
  • Partial application
  • Using operator
  • Using functional
  • Using itertools
  • Using toolz
• Statistical Computing
  • Big O complexity
  • Sorting algorithms
• Scalars
  • Integers
  • Endianess
  • Real numbers
• Vectors
  • Vector space
  • Column vectors
  • Norms and distances
  • Vector operations
  • Dot product
  • Outer product
• Matrices
  • Matrices
  • Matrix representation in Python
  • Properties of a matrix
  • Solving linear equations (Matrix-vector multiplication)
• Sparse Matrices
  • Creating a sparse matrix
  • Compressed Sparse Row and Column formats
  • COO summation convention
  • Solving large sparse linear systems
• Working with Matrices
  • Matrices and liner combinations
  • Permutation matrices
  • Matrix partitioning
• Solving Linear Equations
  • Linear Equations
  • Gaussian elimination
  • Gauss-Jordan elimination
  • Gaussian-Jordan elimination and the number of solutions
  • LU Decomposition
  • Cholesky Decomposition
• Linear least squares
  • Normal equations
  • Projection matrices
  • Optimization
• Applications of Linear Alebra: PCA
  • Variance and covariance
  • Eigendecomposition of the covariance matrix
  • Covariance matrix as a linear transformation
  • PCA
  • Eigendecomposition of the covariance matrix
  • Change of basis via PCA
  • Graphical illustration of change of basis
  • Dimension reduction via PCA
• Understanding the SVD
  • Useful reference
  • Singular value decomposition
  • (1) The matrix A
  • (2) Orthogonal matrices \(U\) and \(V^T\)
  • (3) Diagonal matrix \(S\)
  • Covariance, PCA and SVD
  • PCA
  • Using SVD for PCA
• Linear Algebra Examples
  • Resources
  • Exact solution of linear system of equations
  • Basic information about a matrix
  • Least-squares solution
  • Matrix Decompositions
• Review of linear algebra
  • Coordinates
  • Linear Transformations
  • Matrices, Transformations and Geometric Interpretation
  • Example - Find a Matrix Representation of a Linear Transformation
  • Example - Change to a Different Basis
  • \(m < n\)
  • \(m = n\)
  • \(m > n\)
  • Elementary row and matrix operations
  • Echelon forms
  • Example
• Finding Roots of Equations
  • Calculus review
  • Main Issues in Root Finding in One Dimension
  • Bisection Method
  • Secant Method
  • Newton-Raphson Method
  • Gauss-Newton
  • Inverse Quadratic Interpolation
  • Brentq Method
  • Roots of polynomials
  • Using scipy.optimize
• Numerical Optimization
  • Categorize your optimization problem
  • Sketch of lecture
• Univariate Optimization
  • Interval elimination
• Convexity
• Line Search Methods
• Steepest Descent
• Newton’s Method
• Coordinate Descent
  • Newton CG Algorithm
  • BFGS - Broyden–Fletcher–Goldfarb–Shanno
  • Nelder-Mead Simplex
  • Powell’s Method
• Solvers
  • Levenberg-Marquardt (Damped Least Squares)
  • Newton-Krylov
• GLM Estimation and IRLS
  • Constrained Optimization and Lagrange Multipliers
• Using optimization routines from scipy and statsmodels
  • Using scipy.optimize
  • Some applications of optimization
  • Optimization of standard statistical models
• Line search in gradient and Newton directions
  • Demo functions
  • Gradient descent with step size found by numerical minimization
  • Gradient descent with analytic step size for quadratic function
  • Line search in Newton direction with analytic step size
• Least squares optimization
  • Working with matrices
  • Example
• Gradient Descent Optimizations
  • Smoothing with exponentially weighted averages
  • Momentum in 1D
  • Momentum and RMSprop in 2D
  • Implementing a custom optimization routine for scipy.optimize
• Constrained Optimization
  • Lagrange multipliers and constrained optimization
  • Using scipy.optimize
  • Solve unconstrained problem
  • Solve constrained problem using scipy.optimize
  • Solve constrained problem using Lagrange multipliers
• Parallel Programming
  • Common sense
  • Basic Concepts
  • Embarrassingly parallel programs
• Multi-Core Parallelism
  • Vanilla Python
  • The concurrent.futures module
  • Using processes in parallel with ProcessPoolExecutor
  • Using processes in parallel with ThreadPoolExecutor
  • Using multiprocessing
• Using ipyparallel
  • Starting engines
  • Basic concepts of ipyparallel
  • Using parallel magic commands
• Native code compilation
  • Timing code
  • Profiling
  • Optimizing a function
  • Python
  • JIT with numba
  • Cython
  • Cython with static types
• Just-in-time compilation (JIT)
  • Using numexpr
  • Using numba
  • Using numba vectorize and guvectoize
  • Parallelization with vectorize and guvectorize
• Cython
  • Resources
  • Using Cython annnotations to identify bottlenecks
  • Using Cython cdefs and directives
  • Parallel execution with Cython
  • Speeding up Mandelbrot set visualizations
  • Using Cython wtih ipyparallel
• Introduction to C++
  • Hello world
  • Namespaces
  • Types
  • Type conversions
  • Header, source, and driver files
  • Input and output
  • Arrays
  • Loops
  • Function arguments
  • Arrays, pointers and dynamic memory
  • Functions
  • Generic programming with templates
  • Anonymous functions
  • Function pointers
  • Standard template library (STL)
  • STL algorithms
  • Random numbers
  • Numerics
  • Probability distributions and statistics
• Using pybind11
  • Resources
  • Example 1 - Basic usage
  • Example 2 - Adding doc and named/default arguments
  • Example 5 - Using STL containers
  • Using cppimport
  • Example 6: Vectorizing functions for use with numpy arrays
  • Example 7: Using numpy arrays as function arguments and return values
  • Example 8: More on working with numpy arrays (optional)
  • Example 9: Using the C++ eigen library to calculate matrix inverse and determinant
  • Example 10: Using pybind11 with openmp (optional)
• Random Variables
  • Why are random numbers useful?
  • Where do random numbers in the computer come from?
  • Using numpy.random
  • Sampling with and without replacement
  • Shuffles, permutations and combinations
  • Using scipy.stats
  • Sampling
• Probabilistic Programming Concepts
  • Bayes theorem and parameter estimation
  • Probabilistic Programming
  • Estimating integrals
  • Numerical integration (Quadrature)
  • Curse of dimensionality and concentration of measure
  • Drawing pictures
  • Evaluation
• Monte Carlo Methods
  • Setting the random seed
  • Adjusting p-values for multiple testing
  • Jackknife estimate of parameters
  • Leave one out cross validation (LOOCV)
  • Estimating the CDF
  • Estimating the PDF
• Monte Carlo integration
  • Integrals and averates
  • Simple Monte Carlo integration
  • Intuition behind Monte Carlo integration
  • Intuition for error rate
  • Variance Reduction
• Markov Chains
  • Example: Random walk and diffusion
  • Markov Chain
  • Markov chain averages
• Metropolis and Gibbs Sampling
  • Introduction to MCMC
  • Bayesian Data Analysis
  • Hierarchical models
• Hamiltonian Monte Carlo (HMC)
  • Use of auxiliary variables
  • Hamiltonian Monte Carlo (HMC)
• Using PyMC3
  • Introduction to PyMC3
  • Example: Estimating coin bias
  • Evaluate convergence
  • Saving traces
  • Estimating parameters of a normal distribution
  • Sampling from prior
  • Sampling from posterior
  • Evaluating goodness-of-fit
  • Using a custom likelihood
• Linear regression
  • Plate diagram
  • Setting up and fitting linear model
  • Posterior predictive checks
• Using the GLM module
• Robust linear regression
  • Effect of outlier on linear regression
  • Use a T-distribution for the errors for a more robust fit
• Logistic regression
  • Observed data
• Hierarchical model
  • Hierarchical model
  • Compare the length of the credible interval with the number of observations for each county.
• Mixture models
  • The Categorical distribution
  • The Dirichlet distribution
  • Finite Gaussian Mixture Model
  • Dirichlet process
• Gaussian process models
  • Generative model
  • Simple example of sampling from a GP model
• PyStan
  • Useful links
  • Coin toss
  • Estimating mean and standard deviation of normal distribution
  • Estimating parameters of a linear regression model
  • Simple Logistic model
• Tensorflow
  • Working with tensors
  • Keras
  • Tensorflow proability
  • Regression

Indices and tables

• Index

• Module Index

• Search Page