1
Preamble
2
Introduction
2.1
Data Science
2.2
Predictive Analytics
2.3
Artificial Intelligence
2.4
Machine learning (ML)
2.5
Deep learning
3
Probability and Uncertainty
3.1
Random variables
3.1.1
Conditional Probability
3.2
Expectation
3.3
Variance
3.3.1
Conditional Probability & Independence
3.3.2
Sensitivity and Specificity
3.4
Bayesian Learning
3.4.1
Enigma machine: Code-breaking
3.5
Play the Winner Rule
3.5.1
Black Swans
3.6
Continuous Random Variables
3.6.1
Standard Deviation and Covariance
3.7
Common Distributions
3.7.1
Binomial Distribution
3.7.2
Normal or Gaussian Distribution
3.7.3
Poisson Distribution
4
Linear Regression
4.1
Least Squares Principle
4.1.1
Statistical Properties of Regression
4.1.2
Predictive Uncertainty
4.1.3
Input Transformations
4.1.4
Linear model
4.1.5
Log-Log linear model
4.2
Multiple Regression
4.2.1
Regression Model
4.2.2
Transformation
4.2.3
Interpretation
4.2.4
Prediction
4.3
Interactions
4.3.1
Models with Interactions
4.3.2
Models with Interactions and Dummies
4.4
and finally, consider 3-way interactions
4.5
Dummies
4.5.1
Regression Strategy
4.5.2
Predictive Analytics: General Introduction
5
Logistic Regression
5.1
roc curve and fitted distributions
5.2
a standard `max prob’ (p=.5) rule
5.3
Model Selection
5.4
Small Sample Size
5.4.1
Spike-and-Slab Prior
5.4.2
Horseshoe Prior
5.5
Bayesian Model Selection
6
Optimisation
6.1
Preliminaries
6.2
Newton’s Method
6.2.1
Convergence analysis
6.3
Stochastic Gradient Descent
6.4
Automatic Differentiation (AD)
6.4.1
Computational Graph
6.4.2
Back-Propagation
6.4.3
Backprop for Classification
6.4.4
Convergence analysis
6.4.5
Polyak’s Momentum
6.4.6
SGD vs Newton
6.4.7
Proximal Gradient Methods
6.4.8
Proximal operators and Moreau envelopes
6.4.9
Examples: Proximal Operators
6.4.10
Iterative shrinkage thresholding
6.4.11
Dual ascent
6.4.12
Augmented Lagrangian
6.4.13
Alternating Directions Method of Multipliers (ADMM)
6.4.14
Bregman divergence and exponential families
6.5
Alternating Method of Multipliers (ADMM)
6.6
Markov Chain Monte Carlo (MCMC)
6.6.1
Metropolis-Hastings Algorithm
6.6.2
Independence MH
6.6.3
Random-walk Metropolis
6.6.4
Gibbs Samples
6.6.5
Hybrid chains
6.6.6
Missing values in normal random sample
6.6.7
Stochastic Variational Inference
6.6.8
Auto-Encoding Variational Bayes
6.6.9
Generative Models
6.6.10
Shallow Factor Models
References
Foundations of Data Sceince: Predictive Analytics and AI
Foundations of Data Sceince: Predictive Analytics and AI
Nick Polson and Vadim Sokolov
2022-01-31
Chapter 1
Preamble