Machine Learning CSCI-GA-2565 · Fall 2023 · NYU Courant Computer Science

Instructors Mengye Ren
Lecture Tuesday 4:55pm-6:55pm (Silv 414)

About This Course

This course covers a wide variety of introductory topics in machine learning and statistical modeling, including statistical learning theory, convex optimization, generative and discriminative models, kernel methods, boosting, latent variable models and so on. The primary goal is to provide students with the tools and principles needed to solve the machine learning problems found in practice. Course syllabus can be found here.

For registration information, please contact CS Graduate Office.

Prerequisites

If you'd like to waive the prerequisites, please send an email to the instructor. For each prerequisite, please clearly list which courses you've taken are equivalent, and highlight it in the transcript.

Logistics

Grading

Resources

Related courses

Past exams

  • 2023 Midterm & Solution
  • 2021 Midterm & Solution
  • 2020 Midterm & Solution
  • Textbooks

    The cover of Elements of Statistical Learning The cover of An Introduction to Statistical Learning The cover of Understanding Machine Learning: From Theory to Algorithms The cover of Pattern Recognition and Machine Learning The cover of Bayesian Reasoning and Machine Learning
    The Elements of Statistical Learning (Hastie, Friedman, and Tibshirani)
    This will be our main textbook for L1 and L2 regularization, trees, bagging, random forests, and boosting. It's written by three statisticians who invented many of the techniques discussed. There's an easier version of this book that covers many of the same topics, described below. (Available for free as a PDF.)
    An Introduction to Statistical Learning (James, Witten, Hastie, and Tibshirani)
    This book is written by two of the same authors as The Elements of Statistical Learning. It's much less intense mathematically, and it's good for a lighter introduction to the topics. (Available for free as a PDF.)
    Understanding Machine Learning: From Theory to Algorithms (Shalev-Shwartz and Ben-David)
    Covers a lot of theory that we don't go into, but it would be a good supplemental resource for a more theoretical course, such as Mohri's Foundations of Machine Learning course. (Available for free as a PDF.)
    Pattern Recognition and Machine Learning (Christopher Bishop)
    Our primary reference for probabilistic methods, including bayesian regression, latent variable models, and the EM algorithm. It's highly recommended, but unfortunately not free online.
    Bayesian Reasoning and Machine Learning (David Barber)
    A very nice resource for our topics in probabilistic modeling, and a possible substitute for the Bishop book. Would serve as a good supplemental reference for a more advanced course in probabilistic modeling, such as DS-GA 1005: Inference and Representation (Available for free as a PDF.)
    Hands-On Machine Learning with Scikit-Learn and TensorFlow (Aurélien Géron)
    This is a practical guide to machine learning that corresponds fairly well with the content and level of our course. While most of our homework is about coding ML from scratch with numpy, this book makes heavy use of scikit-learn and TensorFlow. Comfort with the first two chapters of this book would be part of the ideal preparation for this course, and it will also be a handy reference for practical projects and work beyond this course, when you'll want to make use of existing ML packages, rather than rolling your own.
    Data Science for Business (Provost and Fawcett)
    Ideally, this would be everybody's first book on machine learning. The intended audience is both the ML practitioner and the ML product manager. It's full of important core concepts and practical wisdom. The math is so minimal that it's perfect for reading on your phone, and I encourage you to read it in parallel to doing this class, especially if you haven't taken DS-GA 1001.

    Other tutorials and references

    Software

    Lectures

    Week 1

    Topics Materials References

    Lecture Sep 5

    Topics

    Materials

    (None)

    References

    (None)

    Week 2

    Topics Materials References

    Lecture Sep 12

    Topics

    • Gradient descent
    • Stochastic gradient descent
    • Loss functions
    • Slides

    Materials

    References

    (None)

    Week 3

    Topics Materials References

    Lecture Sep 19

    Topics

    • Feature selection
    • Regularization
    • Lasso Optimization
    • Slides

    Materials

    (None)

    References

    (None)

    Week 4

    Topics Materials References

    Lecture Sep 26

    Topics

    • Support Vector Machines
    • Subgradient Descent
    • SVM Dual
    • Slides

    Materials

    References

    (None)

    Week 5

    Topics Materials References

    Lecture Oct 3

    Topics

    • Feature Maps
    • Kernel Trick
    • Representer Theorem
    • Maximum Likelihood Estimation
    • Slides

    Materials

    (None)

    References

    Week 7

    Topics Materials References

    Lecture Oct 17

    Topics

    • Maximum Likelihood Estimation
    • Generative and Discriminative Models
    • Bayes Rule
    • Slides

    Materials

    References

    (None)

    Week 9

    Topics Materials References

    Lecture Oct 31

    Topics

    • Bayesian Methods
    • Bayesian Regression
    • Multi-class
    • Slides

    Materials

    (None)

    References

    (None)

    Week 10

    Topics Materials References

    Lecture Nov 7

    Topics

    • Linear Multiclass Predictors
    • Structured Prediction
    • Decision Trees
    • Slides

    Materials

    References

    Week 11

    Topics Materials References

    Lecture Nov 14

    Topics

    • Random Forests
    • AdaBoost
    • Gradient Boosting
    • Slides

    Materials

    References

    • JWHT 8.1 (Trees)
    • HTF 9.2 (Trees)

    Week 12

    Topics Materials References

    Lecture Nov 21

    Topics

    • Feature Learning
    • Backpropagation
    • Neural Networks
    • Slides

    Materials

    (None)

    References

    Week 13

    Topics Materials References

    Lecture Nov 28

    Topics

    • Deep Learning
    • Transformers
    • Interpretability
    • Adversarial Examples
    • Slides

    Materials

    (None)

    References

    Week 14

    Topics Materials References

    Lecture Dec 5

    Topics

    • k-Means
    • Gaussian Mixture Models
    • Expectiation Maximization
    • Variational Autoencoders
    • Slides

    Materials

    (None)

    References

    Week 15

    Topics Materials References

    Lecture Dec 12

    Topics

    Materials

    (None)

    References

    (None)

    Assignments

    Late Policy: Homeworks are due at 12:00 PM Eastern time (Noon) on the date specified. You have 4 late days in total which can be used throughout the semester without penalty. Once you run out of late days, each additional late day will incur a 20% penalty. For example, if you submit an assignment 1 day late after using all your late days, a score of 90 will only be counted as 72. Note that the maximum late days per homework is two days, meaning that Gradescope will not accept submissions 48 hours after the due date.

    Collaboration Policy: You may form study groups and discuss problems with your classmates. However, you must write up the homework solutions and the code from scratch, without referring to notes from your joint session. In your solution to each problem, you must write down the names of any person with whom you discussed the problem—this will not affect your grade.

    Submission: Homework should be submitted through Gradescope. If you have not used Gradescope before, please watch this short video: "For students: submitting homework." At the beginning of the semester, you will be added to the Gradescope class roster. This will give you access to the course page, and the assignment submission form. To submit assignments, you will need to:

    1. Upload a single PDF document containing all the math, code, plots, and exposition required for each problem.
    2. Where homework assignments are divided into sections, please begin each section on a new page.
    3. You will then select the appropriate page ranges for each homework problem, as described in the "submitting homework" video.

    Feedback: Check Gradescope to get your scores on each individual problem, as well as comments on your answers. Regrading requests should be submitted on Gradescope.

    Homework 0

    Typesetting your homework

    Due: January 1st, 12:00 PM Eastern time (Noon)

    Homework 1

    Linear Regression & Gradient Descent

    Due: October 3rd, 12:00 PM Eastern time (Noon)

    Homework 2

    SVM & Logistic Regression

    Due: October 17th, 12:00 PM Eastern time (Noon)

    Homework 3

    Bayesian ML & Multiclass

    Due: November 14th, 12:00 PM Eastern time (Noon)

    Homework 4

    Decision Tree, Boosting, & Neural Networks

    Due: December 5th, 12:00 PM Eastern time (Noon)

    Course Project

    The final course project constitutes 30% of your overall grade. The objective is to apply the machine learning concepts acquired during this course to a real-world problem. Choose a pertinent and applicable issue, identify an appropriate data source for your machine learning solution, and if no suitable data source exists, propose methods to gather the required data efficiently.

    Template and project instructions package can be found here: pdf zip


    Key dates

    People

    Instructor

    Graders