Machine Learning CSCI-GA-2565 · Fall 2024 · NYU Courant Computer Science

Instructors Mengye Ren
Lecture Tuesday 4:55pm-6:55pm (31 Washington Pl, Room 411)

About This Course

This course covers a wide variety of introductory topics in machine learning and statistical modeling, including statistical learning theory, convex optimization, generative and discriminative models, kernel methods, boosting, latent variable models and so on. The primary goal is to provide students with the tools and principles needed to solve the machine learning problems found in practice. Course syllabus can be found here.

For registration information, please contact CS Graduate Office.

Prerequisites

If you'd like to waive the prerequisites, please send an email to the instructor. For each prerequisite, please clearly list which courses you've taken are equivalent, and highlight it in the transcript.

Logistics

Grading

Resources

Related courses

Past exams

  • 2023 Fall Midterm & Solution
  • 2023 Spring Midterm & Solution
  • Textbooks

    The cover of Elements of Statistical Learning The cover of An Introduction to Statistical Learning The cover of Understanding Machine Learning: From Theory to Algorithms The cover of Pattern Recognition and Machine Learning The cover of Bayesian Reasoning and Machine Learning
    The Elements of Statistical Learning (Hastie, Friedman, and Tibshirani)
    This will be our main textbook for L1 and L2 regularization, trees, bagging, random forests, and boosting. It's written by three statisticians who invented many of the techniques discussed. There's an easier version of this book that covers many of the same topics, described below. (Available for free as a PDF.)
    An Introduction to Statistical Learning (James, Witten, Hastie, and Tibshirani)
    This book is written by two of the same authors as The Elements of Statistical Learning. It's much less intense mathematically, and it's good for a lighter introduction to the topics. (Available for free as a PDF.)
    Understanding Machine Learning: From Theory to Algorithms (Shalev-Shwartz and Ben-David)
    Covers a lot of theory that we don't go into, but it would be a good supplemental resource for a more theoretical course, such as Mohri's Foundations of Machine Learning course. (Available for free as a PDF.)
    Pattern Recognition and Machine Learning (Christopher Bishop)
    Our primary reference for probabilistic methods, including bayesian regression, latent variable models, and the EM algorithm. It's highly recommended, but unfortunately not free online.
    Bayesian Reasoning and Machine Learning (David Barber)
    A very nice resource for our topics in probabilistic modeling, and a possible substitute for the Bishop book. Would serve as a good supplemental reference for a more advanced course in probabilistic modeling, such as DS-GA 1005: Inference and Representation (Available for free as a PDF.)
    Hands-On Machine Learning with Scikit-Learn and TensorFlow (Aurélien Géron)
    This is a practical guide to machine learning that corresponds fairly well with the content and level of our course. While most of our homework is about coding ML from scratch with numpy, this book makes heavy use of scikit-learn and TensorFlow. Comfort with the first two chapters of this book would be part of the ideal preparation for this course, and it will also be a handy reference for practical projects and work beyond this course, when you'll want to make use of existing ML packages, rather than rolling your own.
    Data Science for Business (Provost and Fawcett)
    Ideally, this would be everybody's first book on machine learning. The intended audience is both the ML practitioner and the ML product manager. It's full of important core concepts and practical wisdom. The math is so minimal that it's perfect for reading on your phone, and I encourage you to read it in parallel to doing this class, especially if you haven't taken DS-GA 1001.

    Other tutorials and references

    Software

    Lectures

    Week 1

    Topics Materials References

    Lecture Sep 3

    Topics

    Materials

    (None)

    References

    (None)

    Week 2

    Topics Materials References

    Lecture Sep 10

    Topics

    • Gradient descent
    • Stochastic gradient descent
    • Loss functions
    • Slides
    • Scribble

    Materials

    References

    (None)

    Week 3

    Topics Materials References

    Lecture Sep 17

    Topics

    • Feature selection
    • Regularization
    • Lasso Optimization
    • Max margin classifiers
    • Slides
    • Scribble

    Materials

    (None)

    References

    (None)

    Week 4

    Topics Materials References

    Lecture Sep 24

    Topics

    • Support Vector Machines
    • Subgradient Descent
    • SVM Dual
    • Slides
    • Scribble

    Materials

    References

    (None)

    Week 5

    Topics Materials References

    Lecture Oct 1

    Topics

    Materials

    (None)

    References

    Week 6

    Topics Materials References

    Lecture Oct 8

    Topics

    Materials

    References

    (None)

    Assignments

    Late Policy: Homeworks are due at 12:00 PM Eastern time (Noon) on the date specified. You have 4 late days in total which can be used throughout the semester without penalty. Once you run out of late days, each additional late day will incur a 20% penalty. For example, if you submit an assignment 1 day late after using all your late days, a score of 90 will only be counted as 72. Note that the maximum late days per homework is two days, meaning that Gradescope will not accept submissions 48 hours after the due date.

    Collaboration Policy: You may form study groups and discuss problems with your classmates. However, you must write up the homework solutions and the code from scratch, without referring to notes from your joint session. In your solution to each problem, you must write down the names of any person with whom you discussed the problem—this will not affect your grade.

    Submission: Homework should be submitted through Gradescope. If you have not used Gradescope before, please watch this short video: "For students: submitting homework." At the beginning of the semester, you will be added to the Gradescope class roster. This will give you access to the course page, and the assignment submission form. To submit assignments, you will need to:

    1. Upload a single PDF document containing all the math, code, plots, and exposition required for each problem.
    2. Where homework assignments are divided into sections, please begin each section on a new page.
    3. You will then select the appropriate page ranges for each homework problem, as described in the "submitting homework" video.

    Feedback: Check Gradescope to get your scores on each individual problem, as well as comments on your answers. Regrading requests should be submitted on Gradescope.

    Homework 0

    Typesetting your homework

    Due: January 1st, 12:00 PM Eastern time (Noon)

    Homework 1

    Linear Regression & Gradient Descent

    Due: October 1st, 12:00 PM Eastern time (Noon)

    Homework 2

    SVMs, Kernels & Logistic Regression

    Due: October 15th, 12:00 PM Eastern time (Noon)

    Course Project

    The final course project constitutes 30% of your overall grade. The objective is to apply the machine learning concepts acquired during this course to a real-world problem. Choose a pertinent and applicable issue, identify an appropriate data source for your machine learning solution, and if no suitable data source exists, propose methods to gather the required data efficiently. More project instruction

    Template and detailed project instructions package can be found here: pdf zip


    Key dates

    People

    Instructor

    Graders