CS 6840: Natural Language Processing
Time and Location: Tue, Thu 9:00 – 10:20am, ARC 159
Instructor: Razvan Bunescu
Office: Stocker 341
Office Hours: Tue, Thu 3:00 – 4:00pm, or by email appointment
Email: bunescu @ ohio edu
Recommended Supplementary Text (PDF available online):
Natural Language Processing, by Jacob Eisenstein. 2019.
Speech and Language Processing (3rd edition draft), by Daniel Juraksfy and James E. Martin. 2019.
Natural Language Processing (NLP) is a branch of Artificial Intelligence concerned with developing computer systems that can process or generate natural language. Major applications of NLP include machine translation, sentiment analysis, speech recognition, information retrieval / web search engines, question answering, or information extraction. In this course, students will learn how to use modern machine learning (ML) techniques to solve fundamental NLP tasks, such as training vector-based representations of words and their meanings, document classification, syntactic parsing, language modeling, coreference resolution, entity linking, and semantic parsing.
Students are expected to be comfortable with programming in Python and have basic knowledge of formal languages (regular and context free grammars), linear algebra, probability theory and statistics. Knowledge of deep learning will be very useful, though not strictly necessary as long as the student is willing to learn. Each ML model will be introduced in class and relevant supplemental online materials will be provided throughout the course.
- Syllabus & Introduction
- Text Classification with Perceptron, SVMs, and Logistic Regression
- Gradient Descent Algorithms
- Linear algebra and optimization in Python
- HMMs and Part of Speech Tagging
- Markov Models, chapter 9 in Chris Manning and Hinrich Schutze, Foundations of Statistical Natural Language Processing, MIT Press. Cambridge, MA, May 1999
- Part-of-Speech Tagging Guidelines for the Penn Treebank Project, Beatrice Santorini, Technical Report, 1990
- A Maximum Entropy Model for Part-Of-Speech Tagging, Adwait Ratnaparkhi, EMNLP, 1996
- CRFs and Named Entity Recognition
- Non-linear Classification, Neural Networks, and PyTorch
- Word Embeddings
- Natural Language Processing (Almost) from Scratch, Collobert, Weston, Bottou, Karlen, Kavukcuoglu, and Kuksa, JMLR 2011.
- Distributed Representations of Words and Phrases and their Compositionality, Mikolov, Sutskever, Chen, Corrado, and Dean, NIPS 2013.
- Recurrent Neural Networks for NLP
- RNNs with Attention for Machine Translation
- Convolutional Neural Networks for NLP
- Contextualized Word Embeddings and Pre-training for NLP
- ELMo: Deep contextualized word representations, Peters et al., NAACL 2018.
- Transformer: Attention is all you need, Vaswani et al., NIPS 2017.
- GPT: Improving Language Understanding by Generative Pre-Training, Radford et al., OpenAI 2018.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al., NAACL 2019.
- Energy and Policy Considerations for Deep Learning in NLP, Strubell et al., ACL 2019.
- Syntax and Grammars
- Syntactic Parsing