Date Topic covered Assignments
     
12-Jan Class syllabus
Introduction to IST 597 http://clgiles.ist.psu.edu/IST597/
Introduction to Artificial Intelligence (AI) https://en.wikipedia.org/wiki/Artificial_intelligence
Introduction to Deep Learning https://en.wikipedia.org/wiki/Deep_learning
Computational Complexity Big O, http://en.wikipedia.org/wiki/Big_O_notation
Computational Resources
Assignment 00001
19-Jan Please review chapters 1-4 (all) Deep Learning, https://en.wikipedia.org/wiki/Deep_learning
Machine Learning Fundamentals (all) Unstructured Data, http://en.wikipedia.org/wiki/Unstructured_data
Reading and prsentations assigned (all) Deep Learning, Chapters 1-4
(optional) Artificial Neural Networks, https://en.wikipedia.org/wiki/Artificial_neural_network
Introduction to Keras/TensorFlow https://www.tensorflow.org/learn
Assigment 00001 due (all) Deep Learning, Chapter 5
26-Jan The Multilayer Perceptron & Feedforward Networks (all) Deep Learning, Chapter 6
Architecture & Gradient-Based Learning (optional) "Learning Representations by Back-Propagating Errors" (Nature), https://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html
https://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html
3Blue1Brown series on neural networks https://www.3blue1brown.com/lessons/neural-networks
TensorFlow playground https://playground.tensorflow.org/
Assignment 00010
2-Feb Combating Overparametrization:  Regularization (all) Deep Learning, Chapter 7
Tensorflow examples  Building your first neural network - MLP on mnist/fmnist
Team 11 Stochastic Estimation of the Maximum of a Regression Function.
Team 2 Lessons in Neural Network Training: Overfitting May be Harder than Expected.
Project discussions by team
Assignment 00100
9-Feb Team Presentations: Project Proposal
Student Proposal Order Starting with Team 1
Assignment  00010 due
16-Feb Team Presentations continued; TensorFlow/Keras Examples
23-Feb Optimization (all) Deep Learning, Chapter 8
Team 9 Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping
Team 12 Avoiding overfitting of multilayer perceptrons by training derivatives
Assignment 01000
2-Mar Convolutional Networks (all) Deep Learning, Chapter 9
Team 3 Improving Generalization Performance by Switching from Adam to SGD
Team 6 Adaptive Gradient Methods with Dynamic Bound of Learning Rate
DeepLizard https://deeplizard.com/learn/video/YRhxdVk_sIs
Assignment 00100 due
9-Mar Spring Break
16-Mar Recurrent Neural Networks (all) Deep Learning, Chapter 10
Team 10 Deep Residual Learning for Image Recognition
Team 5 Perturbative Neural Networks
Assigment 01000 Due
Assigment 10000
23-Mar Practical Methodology, Autoencoders (all) Deep Learning, Chapter 11, Chapter 14
(optional) Neural Networks: Tricks of the Trade, 2nd Edition -- "Practical Recommendations for Gradient-Based Training of Deep Architectures", https://link.springer.com/chapter/10.1007/978-3-642-35289-8_26
Team 8 Recurrent Highway Networks
Team 1 HyperNetworks
30-Mar GANs and Transformers Attention Is All You Need
Team 7 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Team 14 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Variational Recurrent Auto-Encoders
Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction 
Assignment 10000 due (all) Deep Learning, Chapter 20
6-Apr Neural Networks with Memory Discrete recurrent neural networks for grammatical inference
Team 4 Higher Order Recurrent Networks and Grammatical Inference
Team 13 Neural Random-Access Machines
Neural Turing Machines
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Theoretical Limitations of Self-Attention in Neural Sequence Models
13-Apr Science & Enginnering Informed Deep Learning Deep Learning In Science, Pierre Baldi
Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next
5 minute update presentations by all teams starting 
in numerical order with team 1 
20-Apr Team Presentations
Team 14, 13, 12, 11, 10, 9, 8 
27-Apr Team Presentations
Teams 7, 6, 5, 4, 3, 2, 1 
2-May Project Reports Due PDF and 2 hard copies - ACM paper format 9pt single space double sided.