CS 523 Multimedia systems   
Applied deep learning: Generative
models for creative intelligence


Spring 2017
Tu 330–6pm
ERF 2068
Prof. Angus Forbes
Office Hours:
W 11am-1pm
SELW 1032

Please make an account on the Spring’17 CS 523 Piazza account: https://piazza.com/uic/spring2017/cs523.

This course explores the creative application of new media technologies, including graphics, visualization, audio, machine learning, and interaction technologies. We will focus especially on the use of generative models with which to simulate, accentuate, and ponder the complex systems in our lives. New media artists have traditionally been involved in exploring the creative, cybernetic potentials of new technologies and their cultural ramifications, using creative outputs to question how our awareness and thinking change when we introduce technology into our lives. We are currently in the midst of an explosion of new ideas related to automatic and interactive information processing. Advances in data mining and machine learning are rapidly finding their way into all corners of our lives. In addition to better understanding the nature of these advances (and learning how to make use of them), we will explore how to think critically about their transformative power. We specifically will focus on a class of machine learning algorithms that learn generative models. Students will become familiar with the TensorFlow library for machine intelligence, as well as any other software tools and languages that will help express and document projects (i.e., video software, Matlab, javascript frameworks, etc). In addition to the completion of weekly readings and assignments, students will be responsible for a series of projects that involve the creation, demonstration, and documentation of a series of three projects. Coursework will be organized around the development of these projects, to be completed individually or in small groups.
Week 1

— Course overview & policies
— Introduction to generative systems
Eulerian Video Magnification; lambda vue
Visual Microphone
— Slides: Week 1


— Read Gardner‘s “Five Minds for the Future”
— Process data of your choice using Eulerian Video Magnification
— Download TensorFlow and associated environment / tools to get it setup on your laptop

Week 2

— Survey of creative ML projects
— Slides: Week 2


— Two short assignments due next week: Assignments 1 and 2
— Make sure to have TensorFlow installed and go through first MNIST database tutorial
— Watch first part of Tensorflow and Deep Learning - Without a PhD for a thorough walkthrough of this tutorial
— Read McDonald’s “A Return to Machine Learning”
— Read Amos’s “Image Completion with Deep Learning in TensorFlow”

Week 3

— Short project presentations
— Overview of neural networks + different types neural networks
— TensorFlow lab
— Slides: Week 3


— The first project is due in two weeks: Project 1
— Read Alammar’s A Visual and Interactive Guide to the Basics of Neural Networks
— Read LeCun, Bengio, and Hinton’s Deep Learning
— Read Gatys, Ecker, and Bethge’s A Neural Algorithm of Artistic Style

Week 4

— Convolutional Neural Networks
— TensorFlow lab
— Slides: Week 4


— Read Karn’s An Intuitive Explanation of Convolutional Neural Networks
— Read Krizhevsky, Sutskever, and Hinton’s ImageNet Classification with Deep Convolutional Neural Networks
— Read Zeiler and Fergus’s Visualizing and Understanding Convolutional Networks


- Xing Li, Kristine Lee, and Arthur Nishimoto
Combining Neural Style Transfer and Color Transfer [repo] [additional images]

- Megan Reid, Caleb Foss, and Vivek Krishnakumar
Painting or Drawing? [repo]

- Hengbin Li
An Implementation of Neural Style [repo]

- Manu Mathew Thomas, Ridhi Rustagi, and Abeer Alsaiari
Neural Style Transfer [repo]

- Kruti Sharma, Hengbin Li, and Shreyas Kulkarni
English to German Translations and English to English Conversations [repo]

Week 6

— Recurrent Neural Networks
— Slides: Week 5


— The second project is due in three weeks: Project 2
— Read Karpathy’s The Unreasonable Effectiveness of Recurrent Neural Networks
— Read Oleh’s Understanding LSTM Networks
— Read Graves’s Generating Sequences With Recurrent Neural Networks
— Read Atken’s Realtime Control of Sequence Generation with RNN Ensembles

Week 7

— Generative Adversarial Nets
— Slides: Week 7

Week 8

— Project 2 updates
— Applications of Generative Adversarial Nets
— Slides: Week 8


Project 2 is due next week


- Manu Mathew Thomas, Abeer Alsaiari, and Ridhi Rustagi
Seq2Seq ChatBot: Chat with Joey! [repo]

- James Lave
Huge Orange Captions [repo]

- Xing Li, Kristine Lee, and Arthur Nishimoto
rule30 RNN Generator [repo]

- Peter Hanula and Sergio Barajas
MadLibs-RNN and MathGen-RNN [repo 1] [repo 2]

- Kruti Sharma, Hengbin Li, and Shreyas Kulkarni
English to German Translations and English to English Conversations [repo]

- Shi Yin and Shiwangi Singh
Chinese Tang Poetry Generator [repo]

- Megan Reid, Caleb Foss, and Vivek Krishnakumar
Generate Text UI [repo]

- Krutarth Joshi, Aayush Kataria, and Dhaval Doshi
ChatBot and Deep Lyrics [repo 1] [repo 2]

Week 10

— Overview of Project 3
— Slides: Week 10


— Read Goodfellow et al.’s Generative Adversarial Nets
— Watch Goodfellow’s talk at AIWTB

Week 11

— Slides: Week 11


— Read Nguyen et al.’s Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
— Read Radford et al.’s Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Week 12

— In-class GAN programming exercises
— Literature survey reports

Week 13

— Report on progress of final project
— Slides: Week 13

Week 14

— Meet with groups to work on final project


— All deliverables for Project 3 are due next week, along with a summary report of all assignments and projects completed during the semester.


- Abeer Alsaiari, Ridhi Rustagi, and Manu Mathew Thomas
De-noising GAN [repo] [video] [write-up]

- Shiwangi Singh and Shi Yin
Outpainting using Context Encoders [repo] [demo] [video]

- Kruti Sharma, Hengbin Li, and Shreyas Kulkarni
Sound Generation Using Deep Convolutional GANs [repo + videos] [write-up]

- Peter Hanula and Sergio Barajas
SRGAN-Magnifier [repo + video] [write-up]

- Megan Reid, Caleb Foss, and Vivek Krishnakumar
Object Detection and Image Inpainting Chrome Extension [repo] [video] [write-up]

- Xing Li, Kristine Lee, and Arthur Nishimoto
GANmons: Sketchable Monsters from Generative Adversarial Networks [repo] [video] [write-up]

- Krutarth Joshi, Aayush Kataria, and Dhaval Doshi
WaveNet as an Image Discriminative Model and a Text Discriminative Model [repo] [write-up]

Required materials & texts
— There are no required texts for this course, but students are encouraged to bcome familiar with a wide range of machine learning techniques. (A list of resources is posted toward the bottom of this webpage.)

— Bring your laptop to class.

— Use a git repo to back-up your files and make your projects/assignments available online.

— Set up a user account on Piazza and join CS 523: https://piazza.com/uic/spring2016/cs523.

Evaluation & requirements
Your final grade will be based on your performance on the group project, evaluations of team member performance (peer evaluation), attendance and participation.

On-time class attendance is mandatory. It is not possible to make up or compensate for missed class sessions. More than two unexcused absences will result in a reduction of the final grade by one letter grade; with every additional unexcused absence, the final grade will drop by an additional grade. Some of the discussions and exercises are done and graded in-class so you must attend class to receive these points.

A = Consistent growth as well as excellent work that consistently goes above and beyond what is required.
B = Above average growth as well as above average work.
C = Average growth as well as average work.
D = Dissatisfactory growth and incomplete work.
E = Dissatisfactory growth, incomplete work, and poor attendance.

The numeric breakdown for the final grade follows:

— Project 1: 10%
— Project 2: 20%
— Project 3: 30%
— Shorter assignments: 20%
— Participation: 20%

No cell phone usage in the lab. No eating during class (coffee or gum, etc, is fine though).

Any individuals with learning disabilities or special needs must make the instructor aware of them prior to the due date of the first major assignment. Those who require accommodations for access and participation in this course must be registered with the UIC Disability Resource Center.

If you wish to observe your religious holiday that is in the conflict with mandatory academic attendance, you must notify the instructor in advance.

You are responsible for understanding what constitutes academic dishonesty. Academic dishonesty is an extremely serious offense. All cases of academic dishonesty will be dealt with in accordance with the policies of the University.

Python resources

Official Python2 and Python3 documentation
Codeacademy's Python tutorials
Learn Python the Hard Way
SciPy, NumPy
TensorFlow, An open-source software library for Machine Intelligence
Keras Deep Learning Library for TensorFlow

Machine learning resources

Awesome TensorFlow
A Visual Introduction to Machine Learning
Flow Machines
Learning to Generate Chairs, Tables, and Cars with Convolutional Networks
How do Convolutional Neural Networks work?
Andrej Karpathy's blog, The Unreasonable Effectiveness of Recurrent Neural Networks
Neural Networks and Deep Learning online textbook
Machine Learning for Software Engineers
CHI 2016 Workshop on Human Centered Machine Learning
Piekniewski's blog, Can A Deep Net See A Cat?, Just How Close Are We To Solving Vision?
Google Research, Google Brain
Hubel and Wiesel’s Visual Cortex Cell Recording, 1959
Visual Analysis for Recurrent Neural Networks
A.I. Experiments with Google
Machine Learning is Fun!, parts 1-6
OpenAI Gym
Overview of GANs, KD Nuggets
Deep Advances in Generative Modeling
Fresh Machine Learning, Sirajology
Machine Learning Recipes
Decoding The Thought Vector
Image Kernels Explained Visually
A Guide to Convolution Arithmetic for Deep Learning
The Great AI Awakening
NIPS papers, Code repos for NIPS’16
2016: The Year AI Got Creative
A Visual and Interactive Guide to the Basics of Neural Networks
TensorFlow in a Nutshell, Pts 1,2, and 3
ML@B blog
List of GAN Papers and Projects
Abusing Generative Adversarial Networks to Make 8-bit Pixel Art
40 Ways Deep Learning is Eating the World
Understanding LSTM Networks
How to build a Recurrent Neural Network in TensorFlow, parts 1 through 7
Conv Nets: A Modular Perspective
Understanding Neural Networks Through Deep Visualization
PixelCNN & PixelRNN in TensorFlow
Minimal ML Algorithms
A Gentle Introduction to Backpropagation
A Step-by-Step Backpropagation Example
Anyone Can Learn To Code an LSTM-RNN in Python
Recurrent Neural Network Tutorial for Artists (p5.js)
Sentdex’s RNN Example in Tensorflow
StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, code
Reddit's Machine Learning channel
Reddit's Machine Learning Questions channel

Creative projects

— Deep Dream: blog, code, gallery
— Neural Photo Editor: article, code, video
— DeepBach: article, code, blog
— Flow Composer: article, AI makes pop music, Composing with AI
— EQ-radio: project page
— Eulerian Video Magnification: project page
— Visual Microphone: project page
— Generating Faces with Deconvolution Networks: blog, code
— Image Compression with Neural Networks: article, blog
— Image Completion with Deep Learning in TensorFlow: blog, code
— Wekinator: project page
— Image Synthesis from Yahoo's open_nsfw: blog,
— Face2Face: Real-time Face Capture and Reenactment of RGB Videos: project page", article
— Nightmare Machine: project page
— AlphaGo: article, project page
— DQN: article, project page, DQN-TensorFlow, video
— WaveNet: article, project page, Tensorflow code
— DeepRhyme: blog
— DeepBeat / DopeLearning: project page, article
— PlaNet: Photo Geolocation with Convolutional Neural Networks: article
— A Neural Algorithm of Artistic Style: article, article, code, Keras code, gallery
— Generating Videos with Scene Dynamics: project page (TinyVideo), article
— Quiver: Interactive Deep Convolutional Networks Features Visualization: project page, code
— One-Shot Generalization in Deep Generative Models: article, video clip, video clip
— Image Inpainting : Overview and Recent Advances: article
— Automated Inference on Criminality using Face Images: article
— Image-to-Image Translation with Conditional Adversarial Nets: project page, article
— Zero-Shot Translation: blog, article
— Generative Visual Manipulation on the Natural Image Manifold: video, project page, article
— Magenta TensorFlow: blog
— Neural Doodle: code, blog, article, article, video
— Four Experiments in Handwriting with a Neural Network: project page + references
— Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks : video, project page, article
— Curated AI: A Literary Magazine Written by Machines, for People: homepage, http://curatedai.com/tags/#deep-gimble-ii
— Traffic Sign Recognition with TensorFlow: blog, BelgiumTS dataset
— Project Dreamcatcher: project page, blog
— Big-Data Poetry: http://bdp.glia.ca/, a crumb of friction milks god
— Plug and Play Generative Networks: project page, article, code
— Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis: article
— Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (DCGANs) : project page, article, code, code, Neural Face
— DRAW: A Recurrent Neural Network For Image Generation: article, video, blog, code, more code

Artist and researcher pages

— Kyle McDonald: A Return to Machine Learning, Style Studies
— Rebecca Fiebrink: website
— Memo Akten: The Dawn of Deep Learning, Algorithmic Decision Making, Machine Bias, Creativity and Diversity, other articles on Medium
— Embodied AudioVisual Interaction: EAVI at Goldsmiths
— Christopher Oleh: website
— Big Picture Group: website
— Jessica Rosenkrantz + Nervous System: talk, studio
— Tom White: Neural Facegrids, Transfers
— Artists and Machine Intelligence: Blog postings

Datasets / APIs

Google Knowledge Graph
Yahoo's open_nsfw
Affectiva Emotion SDK (Affdex)
Norvig’s Ngrams data
Kaggle Academic Machine Learning Competitions
MSRA-CFW: Data Set of Celebrity Faces on the Web
Flickr8k, images with captions
MNIST handwritten digits dataset
UCI Machine Learning Repository datasetsYouTube-BoundingBoxes
An extensive dataset of eye movements during viewing of complex images, MIT Saliency Benchmark datasets
Model Zoo
Google BigQuery Public Datasets
KDnuggets list of public datasets
Wikipedia List of Datasets for Machine Learning Research

Related courses

Parag Mital’s Creative Applications of Deep Learning with TensorFlow
Udacity's Deep Learning by Google
Andrew Ng's Coursera Machine Learning
Philippe Pasquier's Generative Art and Computational Creativity
Brad Knox’s and Cynthia Breazeal’s Interactive Machine Learning
Jeremy Howard’s Practical Deep Learning For Coders
Standford’s Deep Learning Tutorial