OR 750. Bayesain Learning

Department of Systems Engineering and Operations Research
George Mason University
Spring 2021

Course Number: OR 750
Location: Zoom Thu 4:30-7:10pm
Instructor: Vadim Sokolov
Office hours: by Appointment
Prerequisites: Undergraduate Calculus, probability theory, statistics, computer programming skills (ideally R).
HW Logistics: You will submit your HW and projects to BlackBoard

Content and goals

This is a graduate course on Bayesain learning. Although basics will be revisited, the pace will be swift so we can get to advanced topics as quickly as possible. This course details classical Bayesain techniques as well as modern approaches from both statistics and machine learning. We will consider some canonical examples of Bayesain analysis but will concentrate on modern Bayesain techniqes, computation and implementation, as well as modern applications. The course material will emphesize deriving and implementing methods over proving theoretical results.

During weeks 11-15, this class will be run in a seminar mode. A student or the instructor will lead the discussion.

Tentative Schedule/List of Topics

Overview: Conditional Probability, Bayes Rule, Bayesain inference, utility theory, distributions and tranformations
Hierarchical models: Bayesian regression, shrinkage (lasso, horseshoe)
Gaussian Process with applications in Bayesian Optimization – Tuning machine learning algorithms – Engineering model calibration
Algorithms – Markov Chain Monte Carlo (MCMC) – Expectation Maximization (EM) – Variational Bayes
Markov and Hidden Markov Models – Kalman Filter – Particle filter – Dynamic Linear Model – Structural Time Series Models

Grading

Rubric: 30% HW, 10% Discussion, 60 % Final Project. No in-class examination.

Cutoffs: A: 90, B: 80, C: 70, F: < 70

Optional Texts

Pattern Recognition and Machine Learning by Bishop (page)
Machine Learning: a Probabilistic Perspective by Murphy (page)
An Introduction to Bayesian Thinking by Clyde et al. (book)
Introduction to Statistical Thought by Lavine (book)
The Probability and Statistics Cookbook by Vallentin (page)

Papers

Exchange Paradox by Christensen and Utts (pdf)
Inference for Nonconjugate Bayesian Models Using the Gibbs Sampler by Carlin and Polson (pdf)
Data Augmentation for Support Vector Machines by Polson and Scott (pdf)
The horseshoe estimator for sparse signals by Carvalho, Polson, Scott (pdf)
Deep learning: A Bayesian perspective (pdf)
Bayesian regularization: From Tikhonov to horseshoe (pdf)
Spatial Interaction and the Statistical Analysis of Lattice Systems by Besag (jstor)
Particle learning and smoothing by Carlos M Carvalho, Michael S Johannes, Hedibert F Lopes, Nicholas G Polson (pdf)
Bayesian model assessment in factor analysis by Lopes and West (pdf)
Tracking epidemics with Google flu trends data and a state-space SEIR model by Dukic, Lopes and Polson (pdf)
A statistical paradox by Lindley (pdf)
The philosophy of statistics by Lindley (pdf)
The Relevance Vector Machine by Tipping (pdf)
BART: Bayesian additive regression trees by Hugh A. Chipman, Edward I. George, Robert E. McCulloch (arxiv)
Bayesian methods for hidden Markov models: Recursive computing in the 21st century by Scott (pdf)
A modern Bayesian look at the multi‐armed bandit by Scott (pdf)
Bayesian analysis of computer code outputs: A tutorial by O’Hagan (paper)
Bayesian analysis of stochastic volatility models by Jacquier, Polson and Rossi (paper)
Ockham’s razor and Bayesian analysis by Jefferys and Berger (pdf)
Bayesian Learning for Neural Networks by Neal (pdf)
MCMC Using Hamiltonian Dynamics by Neal (pdf)
Slice sampling be Neal (pdf)
Bayesian interpolation by MacKay (pdf)
Bayesian online changepoint detectio by Adams and MacKay ()
The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. by Matthew and Gelman (pdf)

Mason Honor Code

To promote a stronger sense of mutual responsibility, respect, trust, and fairness among all members of the George Mason University community and with the desire for greater academic and personal achievement, we, the student members of the university community, have set forth this honor code: Student members of the George Mason University community pledge not to cheat, plagiarize, steal, or lie in matters related to academic work. Students are responsible for their own work, and students and faculty must take on the responsibility of dealing with violations. The tenet must be a foundation of our university culture.\

All work performed in this course will be subject to Mason’s Honor Code. Students are expected to do their own work in the course. For the group project, students are expected to collaborate with their assigned group members. In papers and project reports, students are expected to write in their own words,

Individuals with Disabilities

The university is committed to providing equal access to employment and educational opportunities for people with disabilities.\

Mason recognizes that individuals with disabilities may need reasonable accommodations to have equally effective opportunities to participate in or benefit from the university educational programs, services, and activities, and have equal employment opportunities. The university will adhere to all applicable federal and state laws, regulations, and guidelines with respect to providing reasonable accommodations as necessary to afford equal employment opportunity and equal access to programs for qualified people with disabilities.\

Applicants for admission and students requesting reasonable accommodations for a disability should call the Office of Disability Services at 703-993-2474. Employees and applicants for employment should call the Office of Equity and Diversity Services at 703-993-8730. Questions regarding reasonable accommodations and discrimination on the basis of disability should be directed to the Americans with Disabilities Act (ADA) coordinator in the Office of Equity and Diversity Services.