SYST/OR 664 CSI 674. Bayesian AI

Spring 2025

Location: Engineering Building 2241
Time: Mondays 4:30 pm - 7:10 pm
Office hours: By appointment
Zoom Link: https://gmu.zoom.us/j/98895158802?pwd=3fGVsYOe332Rw7B2AaZkb2LmzRbXU2.1
HW Logistics: You will submit your HW and projects to BlackBoard

Content and goals

Many artificial intelligence problems involve modeling uncertainty. Bayesian probabilistic models represent uncertainty and dependencies between random variables using probability distributions. You will learn the set of rules of probability and computational algorithms to manipulate these distributions. Bayesian approach enhances the effectiveness of conventional AI techniques. This course summarizes various Bayesian-based models and the standard algorithms used with them, supplemented by instances of their practical use. We will discuss applications in science, engineering, economics, medicine, sport, and law. Students will learn the commonalities and differences between the Bayesian and frequentist approaches to statistical inference, how to approach a statistics problem from the Bayesian perspective, and how to combine data with informed expert judgment soundly to derive useful and policy-relevant conclusions. Assignments focus on applying the methods to practical problems.

List of Topics

Unit 1: Introduction: History of AI, Bayes approach to ML and AI. Probability and Bayes Rule.
Unit 2: Utility and Decision Theory
Unit 3: Bayesian Inference with Conjugate Pairs: Single Parameter Models
Unit 4: Bayesian Hypothesis Tests
Unit 5: Stochastic Processes
Unit 6: Markov Chain Monte Carlo
Unit 7: Bayesian Regression: Linear and Bayesian Trees
Unit 8: Quantile Neural Networks for Reinforcement Learning and Uncertainty Quantification
Unit 9: Bayesian Double Descent and Model Selection: Modern Approach to Bias-Variance Tradeoff
Unit 10: Bayesian Neural Networks and Deep Learning

Schedule

Jan 27: First Class
Feb 17: We will not meet, I will post a video lecture
March 10: No class (spring break)
March 17: Midterm exam
May 5: Last class, project presentations

Assignments

Homework is due 11:59PM on the assigned due date. If you have extenuating circumstances, please contact me in advance, and I will consider giving you additional time to complete the assignment for partial credit. Assignments will be posted here and on Blackboard. Please submit your assignments through Blackboard.

Exams are take home and will be posted on Blackboard.

Prerequisites

The formal listed prerequisite is OR 542 or STAT 544 or STAT 554 or equivalent.

The real prerequisites are:

Experience with elementary data analysis such as scatterplots, histograms, hypothesis tests, confidence intervals, and simple linear regression.
A calculus-based probability course - elementary probability theory, discrete and continuous probability distributions, probability mass and probability density functions, cumulative distribution functions, common parametric models such as the normal, binomial and Poisson distributions.
Experience with a high-level programming language. We will use R, a programming language for data analysis, and STAN, a language for specifying and performing inference with Bayesian models.
Comfort with mathematical notation. We will not do proofs, but you will be expected to be comfortable following and doing mathematical derivations.

Computing

We will use R, a powerful (free) statistical graphics and computing language, and STAN, an open-source, cross-platform engine for Bayesian data analysis that can be accessed from within R. Many of the exercises will require programming. RStudio is an integrated development environment for R. Some students prefer Python to R. You may use your preferred software as long as your solution is clear and I can understand what you did, but the solutions and examples will all be in R.

Grading

Grade based entirely on participation in class, homework assignments, midterm and final project.

Midterm 35% + Final project + 35% + Homework 30%. Scores of each component are normalized to be out of 100. Grades will be posted on Bb. Cut-offs: 97 (A+), 93 (A), 90 (A-), 87 (B+), 82 (B), 79 (B-), 77 (C+), 73 (C), 70 (C-), 67 (D+), 60)

Selected references

Bishop, Christopher M., Pattern Recognition and Machine Learning. Springer, Information Science and Statistics series, 2006.
MacKay David J. C., Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003.
Hoff, Peter D., A First Course in Bayesian Statistical Methods. Springer, 2009.
Computer code is available for most of the examples in the book.
Gelman, A., Carlin, J., Stern, H., Dunson, D. B., Vehtari, A. and Rubin, D., Bayesian Data Analysis (3rd edition). CRC Press, 2013.
Reference tex. This comprehensive text has become the standard reference in Bayesian statistical methods. The hyperlink below contains reviews, exercises, data sets and software.
Marin, Jean-Michel and Robert, Christian, Bayesian Essentials with R (2nd edition). Springer, 2014.
Supplemental text (recommended): This recently published book provides comprehensive coverage of computational Bayesian statistics with a focus on conducting Bayesian analyses of real data sets. The range of topics covered is much more extensive than the Hoff text, and will serve as a useful supplement for readers interested in Bayesian treatment of topics not covered in this course, such as generalized linear models, capture-recapture experiments, time series and image analysis. R code and a solution manual are available.
Lee, Peter, Bayesian Statistics: An Introduction (4th edition), Wiley, 2012. Alternate text: The text by Peter Lee is accessible and may be helpful as an alternative treatment. Again, the hyperlink contains additional information, including exercises, solutions, errata and software.

Additional Resources

Wakefield, J. C., et al. “The evaluation of fibre transfer evidence in forensic science: a case study in statistical modelling.” Journal of the Royal Statistical Society Series C: Applied Statistics 40.3 (1991): 461-476. pdf

Mason Honor Code

To promote a stronger sense of mutual responsibility, respect, trust, and fairness among all members of the George Mason University community and with the desire for greater academic and personal achievement, we, the student members of the university community, have set forth this honor code: Student members of the George Mason University community pledge not to cheat, plagiarize, steal, or lie in matters related to academic work. Students are responsible for their own work, and students and faculty must take on the responsibility of dealing with violations. The tenet must be a foundation of our university culture.

All work performed in this course will be subject to Mason’s Honor Code. Students are expected to do their own work in the course. For the group project, students are expected to collaborate with their assigned group members. In papers and project reports, students are expected to write in their own words,

Individuals with Disabilities

The university is committed to providing equal access to employment and educational opportunities for people with disabilities.

Mason recognizes that individuals with disabilities may need reasonable accommodations to have equally effective opportunities to participate in or benefit from the university educational programs, services, and activities, and have equal employment opportunities. The university will adhere to all applicable federal and state laws, regulations, and guidelines with respect to providing reasonable accommodations as necessary to afford equal employment opportunity and equal access to programs for qualified people with disabilities.

Applicants for admission and students requesting reasonable accommodations for a disability should call the Office of Disability Services at 703-993-2474. Employees and applicants for employment should call the Office of Equity and Diversity Services at 703-993-8730. Questions regarding reasonable accommodations and discrimination on the basis of disability should be directed to the Americans with Disabilities Act (ADA) coordinator in the Office of Equity and Diversity Services.