OR 610. Deep Learning

Department of Systems Engineering and Operations Research
George Mason University
Fall 2020


Course Material

Piazza

Instructor: Vadim Sokolov (vsokolov(at)gmu.edu)
TA: Ali Seyedmazloom (aseyedma (at) masonlive.gmu.edu)
Office hours: By appointment

Datacamp

If you are rusty on Python, I suggest you refresh your skills using Datacamp. Datacamp gave students in this class a free access to all of the courses. If you follow the link above you can get your free access using masonlive email. I also listed some of the Python courses I suggest there.

Schedule (Homework are due on Mondays at noon)

Week 1 (Aug 24): Introduction

• What is a neural network?

• Supervised Learning with Neural Networks

• Why is Deep Learning taking off?

• Linear Model

• Linear classification

Week 2 (Aug 31): Gradient Descent

• Logistic Regression Cost Function

• Gradient Descent

• Derivatives

• More Derivative Examples

• Computation graph

• Derivatives with a Computation Graph

• Logistic Regression Gradient Descent

• Gradient Descent on m Examples

Week 3 (Sep 7): Neural Networks

• HW1 Due

• Neural Networks Overview

• Neural Network Representation

• Computing a Neural Network's Output

• Vectorizing across multiple examples

• Explanation for Vectorized Implementation

• Activation functions

• Why do you need non-linear activation functions?

• Derivatives of activation functions

• Gradient descent for Neural Networks

• Backpropagation

• Random Initialization

• Efficient MLP implementation

• Other matrix derivatives

Week 4 (Sep 14): PyTorch

• HW2 Due (NN)

Week 5 (Sep 21): Optimization and regularization

• Training Neural Networks (a)

• Training Neural Networks (b)

• HW3 due (PyTorch)

Week 6, 7 (Sep 28, Oct 5): CNN

• HW4 Due (Optimisation)

• Lecture 9 of CS231n (video, notes)

Week 8,9 (Oct 12, Oct 19): Recurrent Nets

• Project proposals due

Week 10,11, 12 (Oct 26, Nov 2, Nov 9): Bayesian Neural Networks (Hinton Notes)

• 9d - Introduction to the bayesian approach

• 9e - The bayesian interpretation of weight decay

• 9f - MacKays quick and dirty method of fixing weight costs

• 10a - Why it helps to combine models

• 10b - Mixtures of experts

• 10c - The idea of full bayesian learning

• 10d - Making full bayesian learning practical

• 10e - Dropout an efficient way to combine neural nets

Week 13, 14 (Nov 16, Nov 30): Deep Reinforcement Learning

• Lectures 1-5 by David Silver (videos and slides)

• HW6 due (Bayes)

Week 15 (Dec 7): Final Projects

• Final projects are due.

Data analysis projects

You will work in a team of up to 3 people on a Kaggle-like project and will apply deep learning to solve a prediction or data generation problem. By week 8 of the class you should have a team formed and data set + analysis problem identified. You need to submit a 0.5-1 page description of the data and problem you are trying to solve for my feedback and approval. Proposal has to have names and emails of the team members. Description of data set, problem to be solved and proposed architectures. You will post results of your analysis on the class blog post. The final project will be graded on presentation, writing and analysis. All of the posts based on the presentations are posted here vsokolov.org/or610fall2020

Group Work

Both projects and homework can be done in a groups of size of up to 3 people. You can change groups in between. If you do a homework in a gorup, it means that all of the members of the group do it individually and can consult with each other. You can also do 1 submission per group if you prefer. You can use “group” section of the piazza page to find teammates if you need any. If you need help finding a group, please email me.

Grading

Each hw is 10 points, project is 30.

This is a graduate level course focused on developing deep learning predictive models. We will learn both practical and theoretical aspects of deep learning. We will consider applications in engineering, finance and artificial intelligence. It is targeted towards the students who have completed an introductory courses in statistics and optimization. We will make extensive use of computational tools, such as the Python language, both for illustration in class and in homework problems. The class will consist of 9 lectures given by the instructor on several advanced topics in deep learning. At another 5 lectures students will present on a given topic.

Books

• Goodfellow, Ian, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. Vol. 1. Cambridge: MIT press, 2016.

• Ripley, Brian D. Pattern recognition and neural networks. Cambridge university press, 2007.

• Bishop, Christopher M. Neural networks for pattern recognition. Oxford university press, 1995.

Per Topic Resources

Architectures

• Tuning CNN architecture (blog)

• Sequence to Sequence Learning with Neural Networks (paper)

• Skip RNN (blog and paper)

• Learning the Enigma with Recurrent Neural Networks (blog)

• LSTM blog

• Generative Adversarial Networks (presentation)

• GANs at OpenAI (blog)

• Adaptive Neural Trees (paper)

• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

• Deep Residual Learning for Image Recognition

• Training Very Deep Networks

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

• An intriguing failing of convolutional neural networks and the CoordConv solution

• Attention Is All You Need

• NIPS 2016 Tutorial: Generative Adversarial Networks

• Intuitively Understanding Variational Autoencoders

• WaveNet

• PixelCNN

• Processes https:chrisorm.github.io/NGP.html

Optimization

• BV Book

• Convex Analysis by Rockafellar (1970)

• Nesterov's Lecture

• Nesterov (1983)

• Polyak (1964)

• The Tradeoffs of Large Scale Learning

• Nesterov (2004)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (paper)

• SGD (link)

• Stein Variational Adaptive Importance Sampling

• Bayesian Learning via Stochastic Gradient Langevin Dynamics

• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization

• DARTS: Differentiable Architecture Search (code)

• Neural Architecture Search with Reinforcement Learning (code)

• Regularized Evolution for Image Classifier Architecture Search

• On the importance of initialization and momentum in deep learning

• Flat Minima

• Why Momentum Really Works

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

• Sharp Minima Can Generalize For Deep Nets

• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

• The Marginal Value of Adaptive Gradient Methods in Machine Learning

• Revisiting Nesterov’s Acceleration

Theory

• Polyak, Boris, and Pavel Shcherbakov. “Why does Monte Carlo fail to work properly in high-dimensional optimization problems?.” Journal of Optimization Theory and Applications 173, no. 2 (2017): 612-627. (paper)

• Leni, Pierre-Emmanuel, Yohan D. Fougerolle, and Frédéric Truchetet. “Kolmogorov superposition theorem and its application to multivariate function decompositions and image representation.” In Signal Image Technology and Internet Based Systems, 2008. SITIS’08. IEEE International Conference on, pp. 344-351. IEEE, 2008. (paper)

• Klartag, Bo'az. “A central limit theorem for convex sets.” Inventiones mathematicae 168, no. 1 (2007): 91-131. (paper, slides)

• Sun, Chen, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. “Revisiting unreasonable effectiveness of data in deep learning era.” In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 843-852. IEEE, 2017. (blog)

• Bengio, Yoshua, Aaron Courville, and Pascal Vincent. “Representation learning: A review and new perspectives.” IEEE transactions on pattern analysis and machine intelligence 35, no. 8 (2013): 1798-1828. (paper)

• Braun, Jürgen. “An application of Kolmogorov's superposition theorem to function reconstruction in higher dimensions.” (2009). (dissertation)

• Kolmogorov. “On the Representation of Continuous Functions of Several Variables as Superpositions of Continuous Functions of a Smaller Number of Variables” (paper)

• Arnold. “On functions of three variables” (collection of papers)

• Bianchini, Monica, and Franco Scarselli. “On the complexity of shallow and deep neural network classifiers.” In ESANN. 2014.(paper)

• Girosi, Federico, and Tomaso Poggio. “Representation properties of networks: Kolmogorov's theorem is irrelevant.” Neural Computation 1, no. 4 (1989): 465-469. (paper)

• Kůrková, Věra. “Kolmogorov's theorem and multilayer neural networks.” Neural networks 5, no. 3 (1992): 501-506. (paper)

• Poggio, Tomaso, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao. “Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review.” International Journal of Automation and Computing 14, no. 5 (2017): 503-519. (paper)

• Telgarsky, Matus. “Representation benefits of deep feedforward networks.” arXiv preprint arXiv:1509.08101 (2015). (paper)

• Montufar, Guido F., Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. “On the number of linear regions of deep neural networks.” In Advances in neural information processing systems, pp. 2924-2932. 2014. (paper)

• Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. “Understanding deep learning requires rethinking generalization.” arXiv preprint arXiv:1611.03530 (2016). (paper)

• Lin, Henry W., Max Tegmark, and David Rolnick. “Why does deep and cheap learning work so well?.” Journal of Statistical Physics 168, no. 6 (2017): 1223-1247. (paper)

• Stéphane Mallat 1: Mathematical Mysteries of Deep Neural Networks (video)

• Theory of Deep Learning II: Landscape of the Empirical Risk in Deep Learning

• On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition

• Representation Power of Feed forward Neural Networks

• The Power of Depth for Feedforward Neural Networks

Reinforcement Learning

Model-Ensemble Trust-Region Policy Optimization Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

• The Policy of Truth

• Deep Reinforcement Learning Doesn't Work Yet

Section 5: Bayesian DL

• VAE with a VampPrior (paper)

• Bayesian DL (blog)

• Recognition Networks for Approximate Inference in BN20 Networks (paper)

• Non-linear regression models for Approximate Bayesian Computation (paper)

• DR-ABC: Approximate Bayesian Computation with Kernel-Based Distribution Regression (paper)

• Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation (paper)

• Auto-Encoding Variational Bayes (paper)

• Composing graphical models with neural networks for structured representations and fast inference (paper)

• Yes, but Did It Work?: Evaluating Variational Inference

Practical Tricks

• Averaging

• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks (paper)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (paper)

• Auto-Encoding Variational Bayes (paper)

• Twin Networks: Using the Future as a Regularizer (paper)

• Don't Decay the Learning Rate, Increase the Batch Size (paper)

• DL Tuning (blog)

• Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Other Resources

Additional Reading List

• 50 Years of Data Science by Donoho (paper)

• Deep Learning in Neural Networks: An Overview

• Adversarial Reprogramming of Neural Networks

• Toward theoretical understanding of deep learning

Blogs

• Papers with code link

• Security (blog)

• Unsupervised learning (blog)

• Cybersecurity (paper collection)

• Opt Visualization

Videos

• Deep Energy (blog)

• DL Summer school 2015 (videos)

• DL Representations (blog)

• PyData 2017 (videos)

Other courses with good web presence

• Stanford's CS231n (course page)

• Stanford's STATS385 (course page)

• fast.ai

• Nando de Freitas’ course on machine learning

• UC Berkeley Stat241B (lectures)

• UCUC CSE598 (course page)

• Udacity DRL

Tools

• TensorFlow

• Keras

• TF Playground (Google)

• DL Sony

• SnakeViz (python profiler)

• PyTorch

• TS Stan Examples

• OpenAI Glow

Misc Links

• Pytorch resources (a curated list of tutorials, papers, projects)

• Is artificial intelligence set to become art’s next medium?

Previous instances

• Fall 2017

• Fall 2018