Welcome to STA663-2019’s documentation!

Contents:

• Crash course in Jupyter and Python
  • Class Repository
  • Introduction to Jupyter
  • Introduction to Python
• Text
  • Strings
  • The string module
  • String methods
  • Formatting strings
  • Encodings
  • Using regular expressions
  • Examples
• Scalars
  • Integers
  • Real numbers
• Numbers
  • The ndarray: Vectors, matrices and tenosrs
  • Creating ndarrays
  • Indexing
  • Margins and the axis argument
  • Broadcasting
  • Combining ndarrays
  • Splitting ndarrays
  • Vectorization
• Series
  • Numeric
  • Categories (aka “factors”)
  • With datetime index
  • With datetime values
  • Basic operations
  • dplyr type operations
  • tidyr type operations
• Introduction to pandas
  • Series and Data Frames
  • Creating Data Frames
  • Indexing Data Frames
  • Structure of a Data Frame
  • Selecting, Renaming and Removing Columns
  • Selecting, Renaming and Removing Rows
  • Transforming and Creating Columns
  • Sorting Data Frames
  • Summarizing
  • Split-Apply-Combine
  • Combining Data Frames
  • Fixing common DataFrame issues
  • Reshaping Data Frames
  • Chaining commands
  • Moving between R and Python in Jupyter
• Graphics and Visualization in Python
  • Matplotlib
  • Seaborn
• Functional programming in Python (operator, functional, itertoools, toolz)
  • Pure functions
  • Recursive functions
  • Anonymous functions
  • Lazy evaluation
  • Higher order functions
  • Example: Flattening a nested list
  • Closure
  • Decorators
  • Partial application
  • Using operator
  • Using functional
  • Using itertools
  • Using toolz
• Algorithms and Data Structures
  • Data structures
  • Algorithms
  • Memory usage
  • Timing
• Vectors
  • Vector space
  • Column vectors
  • Norms and distances
  • Vector operations
  • Dot product
  • Outer product
• Matrices
  • Matrices
  • Matrix representation in Python
  • Properties of a matrix
  • Solving linear equations (Matrix-vector multiplication)
• Sparse Matrices
  • Creating a sparse matrix
  • Compressed Sparse Row and Column formats
  • COO summation convention
  • Solving large sparse linear systems
• Working with Matrices
  • Matrices and liner combinations
  • Permutation matrices
  • Matrix partitioning
• Solving Linear Equations
  • Linear Equations
  • Gaussian elimination
  • Gauss-Jordan elimination
  • Gaussian-Jordan elimination and the number of solutions
  • LU Decomposition
  • Cholesky Decomposition
• Linear least squares
  • Normal equations
  • Projection matrices
  • Optimization
• Applications of Linear Alebra: PCA
  • Variance and covariance
  • Eigendecomposition of the covariance matrix
  • Covariance matrix as a linear transformation
  • PCA
  • Eigendecomposition of the covariance matrix
  • Change of basis via PCA
  • Graphical illustration of change of basis
  • Dimension reduction via PCA
• Understanding the SVD
  • Useful reference
  • Singular value decomposition
  • (1) The matrix A
  • (2) Orthogonal matrices \(U\) and \(V^T\)
  • (3) Diagonal matrix \(S\)
  • Covariance, PCA and SVD
  • PCA
  • Using SVD for PCA
• Linear Algebra Examples
  • Resources
  • Exact solution of linear system of equations
  • Basic information about a matrix
  • Least-squares solution
  • Matrix Decompositions
• Review of linear algebra
  • Coordinates
  • Linear Transformations
  • Matrices, Transformations and Geometric Interpretation
  • Example - Find a Matrix Representation of a Linear Transformation
  • Example - Change to a Different Basis
  • \(m < n\)
  • \(m = n\)
  • \(m > n\)
  • Elementary row and matrix operations
  • Echelon forms
  • Example
• Finding Roots of Equations
  • Calculus review
  • Main Issues in Root Finding in One Dimension
  • Bisection Method
  • Secant Method
  • Newton-Raphson Method
  • Gauss-Newton
  • Inverse Quadratic Interpolation
  • Brentq Method
  • Roots of polynomials
  • Using scipy.optimize
• Numerical Optimization
  • Categorize your optimization problem
  • Sketch of lecture
• Algorithms for Optimization and Root Finding for Multivariate Problems
  • Convexity
  • Line Search Methods
  • Steepest Descent
  • Newton’s Method
  • Coordinate Descent
  • Solvers
  • GLM Estimation and IRLS
• Using optimization routines from scipy and statsmodels
  • Using scipy.optimize
  • Some applications of optimization
  • Optimization of standard statistical models
• Line search in gradient and Newton directions
  • Demo functions
  • Gradient descent with step size found by numerical minimization
  • Gradient descent with analytic step size for quadratic function
  • Line search in Newton direction with analytic step size
• Least squares optimization
  • Working with matrices
  • Example
• Gradient Descent Optimizations
  • Smoothing with exponentially weighted averages
  • Momentum in 1D
  • Momentum and RMSprop in 2D
  • Implementing a custom optimization routine for scipy.optimize
• Constrained Optimization
  • Lagrange multipliers and constrained optimization
  • Using scipy.optimize
  • Solve unconstrained problem
  • Solve constrained problem using scipy.optimize
  • Solve constrained problem using Lagrange multipliers
• Dimension Reduction
  • Projection when you have an orthogonal basis
  • PCA from scratch
  • Geometry of PCA
  • Projection
  • Algebra of PCA
  • Geometry of SVD
• Nonlinear Dimension Reduction
  • Limitations of PCA
  • Multi-dimensional scaling (MDS)
  • t-distributed Stochastic Neighbor Embedding (t-SNE)
• Clustering
  • Toy example to illustrate concepts
  • How to cluster
  • Model evaluation
  • Selecting models
  • Aligning clusters across samples
• C++ Mechanics
  • Basic C++ program
  • C equivalent
  • Using qualified names
  • Using namespace
  • Using a function
  • Separate compilation
  • Command line arguments
• C Crash Course
  • A tutorial example - coding a Fibonacci function in C
  • C Basics
• Introduction to C++
  • Hello world
  • Namespaces
  • Types
  • Type conversions
  • Command line inputs
  • Header, implementation and driver files
  • Using make
  • A more flexible Makefile
  • Input and output
  • Arrays
  • Loops
  • Function arguments
  • Arrays, pointers and dynamic memory
  • Functions
  • Generic programming with templates
  • Anonymous functions
  • Function pointers
  • Standard template library (STL)
  • STL algorithms
  • Random numbers
  • Statistics
  • Numerics
• Introduction to C++
  • Compilation
  • Execution
  • Compilation
  • Execution
  • Header file(s)
  • Implementation file(s)
  • Driver program
  • Compilation
  • Execution
  • Compilation
  • Execution
  • Compilation
  • Execution
  • Check SVD
• Using pybind11
  • Resources
  • Example 1 - Basic usage
  • Example 2 - Adding doc and named/default arguments
  • Example 5 - Using STL containers
  • Using cppimport
  • Example 6: Vectorizing functions for use with numpy arrays
  • Example 7: Using numpy arrays as function arguments and return values
  • Example 8: More on working with numpy arrays (optional)
  • Example 9: Using the C++ eigen library to calculate matrix inverse and determinant
  • Example 10: Using pybind11 with openmp (optional)
• Cython
  • Resources
  • Using Cython annnotations to identify bottlenecks
  • Using Cython cdefs and directives
  • Parallel execution with Cython
• Just-in-time compilation (JIT)
  • Using numexpr
  • Using numba
  • Using numba vectorize and guvectoize
  • Parallelization with vectorize and guvectorize
• Native code compilation
  • Timing code
  • Profiling
  • Optimizing a function
  • Python
  • JIT with numba
  • Cython
  • Cython with static types
  • Wrapping C++ cdoe
• Pythran and Transonic
• Probabilistic Programming Concepts
  • Bayes theorem and parameter estimation
  • Probabilistic Programming
  • Estimating integrals
  • Numerical integration (Quadrature)
  • Curse of dimensionality and concentration of measure
  • Working with random variables
  • Monte Carlo methods
  • Monte Carlo integration
  • Drawing pictures
• Random Variables
  • Using numpy.random
  • Sampling with and without replacement
  • Shuffles, permutations and combinations
  • Using scipy.stats
  • Sampling
• Monte Carlo Methods
  • Setting the random seed
  • Adjusting p-values for multiple testing
  • Jackknife estimate of parameters
  • Leave one out cross validation (LOOCV)
  • Estimating the CDF
  • Estimating the PDF
• Monte Carlo integration
  • Integrals and averates
  • Simple Monte Carlo integration
  • Intuition behind Monte Carlo integration
  • Intuition for error rate
  • Variance Reduction
• Markov Chains
  • Example: Random walk and diffusion
  • Markov Chain
  • Markov chain averages
• Metropolis and Gibbs Sampling
  • Introduction to MCMC
  • Bayesian Data Analysis
  • Hierarchical models
• Hamiltonian Monte Carlo (HMC)
  • Use of auxiliary variables
  • Hamiltonian Monte Carlo (HMC)
• Deterministic methods
  • Point estimates
  • Laplace approximation
  • Entropy
  • Kullback-Leibler divergence (relative entropy)
  • Evidence lower bound (ELBO)
  • ADVI
• Probabilistic Programming Module
  • 1. Probabilistic Programming background
  • 2. Metropolis, Metropolis-Hastings and Gibbs
  • 3. HMC and NUTS
  • 4. Deterministic inference engines
  • 5. PyMC3 I
  • 6. PyMC3 II
  • 7. PyMC3 III
  • 8. TF
  • 9. TF Probability
• Using PyMC3
  • Introduction to PyMC3
  • Example: Estimating coin bias
  • Evaluate convergence
  • Saving traces
  • Sampling from prior
  • Sampling from posterior
  • Evaluating goodness-of-fit
  • Using a custom likelihood
• Linear regression
  • Plate diagram
  • Setting up and fitting linear model
  • Posterior predictive checks
• Using the GLM module
• Robust linear regression
  • Effect of outlier on linear regression
  • Use a T-distribution for the errors for a more robust fit
• Logistic regression
  • Note that NUTS performs well despite the correlation between height and weight
• Logistic model not using glm
  • Observed data
• Hierarchical model
  • Hierarchical model
  • Compare the length of the credible interval with the number of observations for each county.
• Mixture models
  • The Categorical distribution
  • The Dirichlet distribution
  • Finite Gaussian Mixture Model
  • Dirichlet process
• Gaussian process models
  • Generative model
  • Simple example of sampling from a GP model
• PyStan
  • Useful links
  • Coin toss
  • Estimating mean and standard deviation of normal distribution
  • Estimating parameters of a linear regression model
  • Simple Logistic model
• TensorFlow and Edward
  • Packages
  • References
  • TensorFlow
  • TensorFlow Examples
  • Edward
  • Edward examples
• Using TensorFlow
  • Check Hardware Availability
  • CPU vs GPU
  • Basic computations in TensorFlow
  • Opearations and graphs
  • Reductions
  • Data Flow Graphs
  • Constants, Variables and Placeholders
• Tensorflow
  • Working with tensors
  • Keras
  • Tensorflow proability
  • Regression

Indices and tables

• Index

• Module Index

• Search Page