This course teaches students how to design new products and services that leverage the capabilities of AI and machine learning to improve the quality of peoples lives. Students will learn to follow a matchmaking design, user-centered design, and service design process. Students will learn to ideate; reframing problematic situations by envisioning many possible products and services. Students will learn to iteratively refine and assess their ideas with real users/customers. Class projects will focus on the challenges of deploying systems that generate errors and the challenges of situating intelligent systems such that they harmonize the best qualities of human and machine intelligence.

Artificial Intelligence is inspired by human intelligence, made powerful by human data, and ultimately only useful in how it positively affects the human experience. This course is an introduction to harnessing the power of AI so that it is beneficial and useful to people. We will cover a number of general topics: agency and initiative, AI and ethics, bias and transparency, confidence and errors, human augmentation and amplification, trust and explainability, mixed-initiative systems, and programming by example. These topics will be explored via projects in dialog and speech-controlled systems, automatic speech recognition, computer vision, data science, recommender systems, text summarization, learning science, UI personalization, and visualization. Students will complete individual weekly mini-projects in which they will design and build AI systems across a wide variety of domains. Students should be comfortable with programming; assignments will be primarily in Python and Javascript. Prior experience with AI/machine learning will be useful but is not required. Students will also be responsible for weekly readings and occasional presentations to the class.

Why are things so hard to use these days? Why doesn't this thing I just bought work? Why is this web site so hard to use? These are frustrations that we have all faced from systems not designed with people in mind. The question this course will focus on is: how can we design human-centered systems that people find useful and usable? This course is a broad introduction to designing, prototyping, and evaluating user interfaces. If you take only one course in Human-Computer Interaction, this is the course for you. We will cover theory as well as practical application of ideas from Human-Computer Interaction. Coursework includes lectures, class discussion, homework, class presentations, and group projects. This class is open to all undergrads and grad students, with either technical or non-technical majors. However, there is a programming prerequisite.

The field of human-robot interaction (HRI) is fast becoming a significant area of research in robotics. The basic objective is to create natural and effective interactions between people and robots. HRI is highly interdisciplinary, bringing together methodologies and techniques from robotics, artificial intelligence, human-computer interaction, psychology, education, and other fields. This course is primarily lecture-based, with in-class participatory mini-projects, homework assignments, a group term project that will enable students to put theory to practice, and a final. The topics covered will include technologies that enable human-robot interactions, the psychology of interaction between people and robots, how to design and conduct HRI studies, and real-world applications such as assistive robots.

Planning is one of the core components that enable robots to be autonomous. Robot planning is responsible for deciding in real-time what should the robot do next, how to do it, where should the robot move next and how to move there. This class does an in-depth study of popular planning techniques in robotics and examines their use in ground and aerial robots, humanoids, mobile manipulation platforms and multi-robot systems. The students learn the theory of these methods and also implement them in a series of programming-based projects. To take the class students should have taken an Intro to Robotics class and have a good knowledge of programming and data structures.

This course is an introduction to the theory and algorithms of multirotor vehicle autonomy. Students will work individually to develop a multirotor simulator in Python and C++, add sensors, plan, and perform exploration. Lectures will cover topics to advance the capabilities of the simulator. Homeworks will be designed to increase the autonomy capabilities of the multirotor vehicle. The class will culminate in an individual project that pushes the autonomy capabilities developed throughout the course and may cover multi-robot aerial autonomy, dynamic environment modeling, or advanced planning and control. In order to succeed in the course, students must have a 2nd year science/engineering level background in mathematics (matrices, vectors, coordinate systems) and have already mastered at least one object-oriented programming language like C++ or Python. When the course is over, students will have written a single software system that has been incrementally extended in functionality and regularly debugged throughout the semester.

Foundations and principles of robotic kinematics. Topics include transformations, forward kinematics, inverse kinematics, differential kinematics (Jacobians), manipulability, and basic equations of motion. Course also include programming on robot arms.

Computational neuroscience is an interdisciplinary science that seeks to understand how the brain computes to achieve natural intelligence. It seeks to understand the computational principles and mechanisms of intelligent behaviors and mental abilities and #8212; such as perception, language, motor control, and learning and #8212; by building artificial systems and computational models with the same capabilities. This course explores how neurons encode and process information, adapt and learn, communicate, cooperate, compete and compute at the individual level as well as at the levels of networks and systems. It will introduce basic concepts in computational modeling, information theory, signal processing, system analysis, statistical and probabilistic inference. Concrete examples will be drawn from the visual system and the motor systems, and studied from computational, psychological and biological perspectives. Students will learn to perform computational experiments using Matlab and quantitative studies of neurons and neuronal networks.

Autonomous agents use perception, cognition, actuation, and learning to reliably achieve desired goals, where the agents can be smart homes, mobile robots, intelligent factories, self-driving cars, etc. The goal of this course is to introduce students to techniques needed for developing complete, integrated AI-based autonomous agents. Topics include architectures for intelligent agents; autonomous behaviors, perception, and execution; reasoning under uncertainty; optimization; execution monitoring; machine learning; scheduling; and explanation. A focus of the course will be on the integration and testing of autonomous systems to achieve reliable and robust behavior in the face of sensor noise and uncertainty. The course is project-oriented where small teams of students will design, implement, and evaluate agents that can grow plants autonomously, without human intervention.

This course will explore the future of robot toys by analyzing and programming Anki Cozmo, a new robot with built-in artificial intelligence algorithms. Como is distinguished from earlier consumer robots by its reliance on vision as the primary sensing mode and its sophisticated use of A.I. Its capabilities include face and object recognition, map building, path planning, and object pushing and stacking. Although marketed as a pre-programmed children's toy, Cozmo's open source Python SDK allows anyone to develop new software for it, which means it can also be used for robotics education and research. The course will cover robot software architecture, human-robot interaction, perception, and planning algorithms for navigation and manipulation. Prior robotics experience is not required, just strong programming skills.

In this course, we will first cover the biological and psychological foundational knowledge of biological perceptual systems, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. The course will focus on vision this year, but will also touch upon other sensory modalities. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. Topics include perceptual representation and inference, perceptual organization, perceptual constancy, object recognition, learning and scene analysis.

This course covers the fundamentals of vision cameras and other sensors - how they function, how they are built, and how to use them effectively. The course presents a journey through the fascinating five hundered year history of 'camera-making' from the early 1500's 'camera obscura' through the advent of film and lenses, to today's mirror-based and solid state devices (CCD, CMOS). The course includes a significant hands-on component where students learn how to use the sensors and understand, model and deal with the uncertainty (noise) in their measurements. While the first half of the course deals with conventional 'single viewpoint' or 'perspective' cameras, the second half of the course covers much more recent 'multi-viewpoint' or 'multi-perspective' cameras that includes a host of lenses and mirrors.

This course studies the theory, design, and implementation of text-based search engines. The core components include statistical characteristics of text, representation of information needs and documents, several important retrieval models, and experimental evaluation. The course also covers common elements of commercial search engines, for example, integration of diverse search engines into a single search service ("federated search", "vertical search"), personalized search results, diverse search results, and sponsored search. The software architecture components include design and implementation of large-scale, distributed search engines.

Computational photography is the convergence of computer graphics, computer vision and imaging. Its role is to overcome the limitations of the traditional camera, by combining imaging and computation to enable new and enhanced ways of capturing, representing, and interacting with the physical world. This advanced undergraduate course provides a comprehensive overview of the state of the art in computational photography. At the start of the course, we will study modern image processing pipelines, including those encountered on mobile phone and DSLR cameras, and advanced image and video editing algorithms. Then we will proceed to learn about the physical and computational aspects of tasks such as 3D scanning, coded photography, lightfield imaging, time-of-flight imaging, VR/AR displays, and computational light transport. Near the end of the course, we will discuss active research topics, such as creating cameras that capture video at the speed of light, cameras that look around walls, or cameras that can see through tissue. The course has a strong hands-on component, in the form of seven homework assignments and a final project. In the homework assignments, students will have the opportunity to implement many of the techniques covered in the class, by both acquiring their own images of indoor and outdoor scenes and developing the computational tools needed to extract information from them. For their final projects, students will have the choice to use modern sensors provided by the instructors (lightfield cameras, time-of-flight cameras, depth sensors, structured light systems, etc.). This course requires familarity with linear algebra, calculus, programming, and doing computations with images. The course does not require prior experience with photography or imaging.

Speech Processing offers a practical and theoretical understanding of how human speech can be processed by computers. It covers speech recognition, speech synthesis and spoken dialog systems. The course involves practicals where the student will build working speech recognition systems, speech synthesis systems and integrate them to create a speech interface. This work will be based on existing toolkits. Details of algorithms, techniques and limitations of state of the art speech systems will also be presented. This course is designed for students wishing understand how to process real data for real applications, applying statistical and machine learning techniques as well as working with limitations in the technology.

This course brings together many disciplines of Artificial Intelligence (including computer vision, robot control, reinforcement learning, language understanding) to show how to develop intelligent agents that can learn to sense the world and learn to act by imitating others, maximizing sparse rewards, and/or satisfying their curiosity.

Large datasets pose difficulties across the machine learning pipeline. They are difficult to visualize and introduce computational, storage, and communication bottlenecks during data pre-processing and model training. Moreover, high capacity models often used in conjunction with large datasets introduce additional computational and storage hurdles during model training and inference. This course is intended to provide a student with the mathematical, algorithmic, and practical knowledge of issues involving learning with large datasets. Among the topics considered are: data cleaning, visualization, and pre-processing at scale; principles of parallel and distributed computing for machine learning; techniques for scalable deep learning; analysis of programs in terms of memory, computation, and (for parallel methods) communication complexity; and methods for low-latency inference. The class will include programming and written assignments to provide hands-on experience applying machine learning at scale. An introductory machine learning course is a prerequisite. A strong background in programming will also be necessary; suggested prerequisites include 15-210, 15-214, or equivalent. Students are expected to be familiar with Python or learn it during the course.

The goal of this course is to provide students an understanding and overview of the 'full stack' of deep learning systems, ranging from the high-level modeling design of modern deep learning systems, to the basic implementation of automatic differentiation tools, to the underlying device-level implementation of efficient algorithms. Throughout the course, students will design and build from scratch a complete deep learning library, capable of efficient GPU-based operations, automatic differentiation of all implemented functions, and the necessary modules to support parameterized layers, loss functions, data loaders, and optimizers. Using these tools, students will then build several state-of-the-art modeling methods, including convolutional networks for image classification and segmentation, recurrent networks and self-attention models for sequential tasks such as language modeling, and generative models for image generation.

Building intelligent machines that are capable of extracting meaningful representations from data lies at the core of solving many AI related tasks. In the past decade, researchers across many communities, from applied statistics to engineering, computer science and neuroscience, have developed deep models that are composed of several layers of nonlinear processing. An important property of these models is that they can learn useful representations by re-using and combining intermediate concepts, allowing these models to be successfully applied in a wide variety of domains, including visual object recognition, information retrieval, natural language processing, and speech perception. The goal of this course is to introduce students to both the foundational ideas and the recent advances in deep learning. The first part of the course will focus on supervised learning, including neural networks, back-propagation algorithm, convolutional models, recurrent neural networks, and their extensions with applications to image recognition, video analysis, and language modelling. The second part of the course will cover unsupervised learning, including variational autoencoders, sparse-coding, Boltzmann machines, and generative adversarial networks. This course will assume a reasonable degree of mathematical maturity and will require strong programming skills.

A key challenge in machine learning is that of structured prediction: taking unstructured data as input and producing a structured output. Structured prediction problems abound throughout application areas such as natural language processing, speech processing, computational biology, computer vision, healthcare, and many others. In this course, we will study modern approaches to structured prediction building on probabilistic graphical models, deep learning, and search. The course will focus on three key aspects: models, inference, and learning. The models we consider will focus on both generative and discriminative models such as Bayesian networks, Markov random fields (MRFs), conditional random fields (CRFs), and deep neural networks including convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and #8212; as well as hybrids of graphical models and neural networks. The course will explore approaches to exact and approximate inference: junction tree algorithm, approximate marginal inference by Markov chain Monte Carlo (MCMC) and variational methods, approximate MAP inference by integer linear programming (ILP) and search. We will explore unsupervised, semi-supervised, and supervised learning using different formulations of the learning problem: MLE, Bayesian inference, structured perceptron, M3Ns, learning to search, and autoencoders. Covered applications will include machine translation, speech recognition, DNA sequence analysis, scene understanding, medical diagnosis. This course is cross-listed as 10-418 and 10-618; students registered for 10-618 will do a course project.

In the past decades, researchers have discovered a number of important and deep connections between machine learning theory and algorithmic game theory. This course will explore these connections, both introducing fundamental topics in each area and describing how ideas from each area can shed light on the other.

This course provides a comprehensive introduction to the theory and implementation of algorithms for organizing and searching large text collections. The first half of the course studies text search engines for enterprise and Web environments; the open-source Indri search engine is used as a working example. The second half studies text mining techniques such as clustering, categorization, and information extraction. Programming assignments give hands-on experience with document ranking algorithms, categorizing documents into browsing hierarchies, and related topics.

Neural networks have increasingly taken over various AI tasks, and currently produce the state of the art in many AI tasks ranging from computer vision and planning for self-driving cars to playing computer games. Basic knowledge of NNs, known currently in the popular literature as 'deep learning', familiarity with various formalisms, and knowledge of tools, is now an essential requirement for any researcher or developer in most AI and NLP fields. This course is a broad introduction to the field of neural networks and their 'deep' learning formalisms. The course traces some of the development of neural network theory and design through time, leading quickly to a discussion of various network formalisms, including simple feedforward, convolutional, recurrent, and probabilistic formalisms, the rationale behind their development, and challenges behind learning such networks and various proposed solutions. We subsequently cover various extensions and models that enable their application to various tasks such as computer vision, speech recognition, machine translation and playing games. Instruction Unlike prior editions of 11-785, the instruction will primarily be through instructor lectures, and the occasional guest lecture. Evaluation Students will be evaluated based on weekly continuous-evaluation tests, and their performance in assignments and a final course project. There will be six hands-on assignments, requiring both low-level coding and toolkit-based implementation of neural networks, covering basic MLP, convolutional and recurrent formalisms, as well as one or more advanced tasks, in addition to the final project.

This course introduces modern methods of data analysis, building on the theory and application of linear models from 36-401. Topics include nonlinear regression, nonparametric smoothing, density estimation, generalized linear and generalized additive models, simulation and predictive model-checking, cross-validation, bootstrap uncertainty estimation, multivariate methods including factor analysis and mixture models, and graphical models and causal inference. Students will analyze real-world data from a range of fields, coding small programs and writing reports.