Computer Science 291A
Introduction to Pattern Recognition, Articial Neural Networks, and Machine Learning
|Lecture: TH 9:00am - 10:50am
||Room: PHELP 2510
|Instructor: Yuan-Fang Wang
||Office: Harold Frank Hall, 3113
|Office hours: TH 11:00am - 12:00pm
|Enrollment code: 53322
There is no undergraduate pre-requisite
(though some material covered in CS165A and CS165B might overlap).
This is a first graduate-level introductory CS
course in PR, ANN and ML. Students should have graduate standing
and good preparation in calculus, linear algebra, and
This course will cover several topics on pattern recognition (PR),
artificial neural networks (ANN), and machine learning (ML). Pattern
recognition is a classical research area that deals with recognizing
patterns (objects) based on their features (traits or appearance). It
has seen wide applications in speech recognition, image analysis, target
detection, optical character recognition, fingerprint identification,
dating services, insurance fraud detection, DNA sequence alignment,
protein structure matching, data mining, network intrusion detection,
engine trouble shooting, among many others. Artificial neural networks
provide a general computing framework which is purported to be highly
parallel, distributed, and fault tolerant. Machine learning
deals with algorithms and formulations that enable a machine to learn
and improve its performance from experience, that is, to modify
its behaviors and execution on the basis of acquired information
and data analysis. These areas share a
lot of commonality in addressing similar problems in classification
Though active research activities are still being conducted in these
areas, many useful results and practical algorithms have been developed
which should prove useful. This course will discuss some of these
results. You should consider concepts and algorithms presented here
as general mathematic tools which hopefully will enrich your math
"toolbox" and may become handy some day.
Class attendance and participation in the discussion = 10%
Two - three assignments or miniprojects (written + programming) = 60%
Class/research project presentations = 30%
There is no exam in the course. The final grade will be determined based
on attendance and class participation, assignments, and research/project
presentations. The difference between assignments (miniprojects) and class/research
projects is if you get to choose the topics or work on assigned topics.
Be warned that how much you get out of this course will
in a very large degree be decided by your motivation to learn and your
curiosity to explore and experiment with the many ideas and algorithms
presented in the course.
We will divide the courses into three parts: The first half will focus
on many "traditional", tried-and-true techniques and algorithms in
pattern recognition and artificial neural networks. The second part will
focus on more advanced techniques in machine learning, in particular,
in Support Vector Machines, Boosting, and Kernel Methods. The third part will
be on recent development of "deep" neural networks, particularly their applications to
image and vision problems.
We will discuss the following "traditional" topics (Topics may be
deleted and additional topics added depending on the interests of the
participants and the available time):
We will also discuss the following more advanced topics in
- Introduction to Pattern Recognition
- Parameter Estimation Techniques
- Non-parametric Estimation Techniques
- Linear Discriminant Functions
- Dimension Reduction Techniques
- Nonmetric Data
- Introduction to Artificial Neural Networks
- Simple Perceptrons
- Multi-layer Perceptrons
- Radial Basis Function Networks
- Mixture Density
- Unsupervised Clustering
- Unsupervised Hebbian and Competitive Learning
- Hopfield network
- Correlated and Time-Varying Events (Hidden Markov Models)
Caveat: Based on past experience, we may not
be able to cover all the topics listed above. Even though a fair amount of
mathematics will be introduced, this is not intended to be a course in
mathematics. The purpose of the course is to introduce you to some
practical, useful concepts, designs, and algorithms in PR, ANN, and ML with
a minimum amount of mathematic rigor. For example, we do not prove many
of the results, instead, we provide intuition as to why things turn out
the way they are.
- Support Vector Machines
- Generalization Performance
- Kernel Methods
- Deep learning
- Deep learning applications to image and vision problems
I will lecture using my own notes.
We do not follow any book in any particular order. So none of these books is required.
You can probably find PDF versions of many of these books online.
T. Hastie, R. Tibshirani, and J. H. Friedman, "The Elements of Statistical Learning", 2nd edition, Springer, 2009
Richard O. Duda, Peter E. Hart, and David G.Stork,
"Pattern Classification", second edition, Wiley-Interscience, 2001
"Pattern Recognition and Machine Learning,", Springer,2006.
Tom M. Mitchell,
"Machine Learning", McGraw-Hill, 1997
"Neural Networks", second edition, Prentice Hall, 1999.
John Shawe-Taylor and Nello Cristianini,
"Kernel Methods for Pattern Analysis", Cambridge University Press, 2004
Other Useful References (oldies but goodies)
- Duda and Heart, Pattern Classification and Scene
Analysis, John-Wiley & sons, New York, NY, 1973.
- Fukunaga, Introduction to Statistical Pattern
Recognition, Academic Press, New York, NY, 1972.
General class policies and announcements:
This class has a Web site at
http://www.cs.ucsb.edu/~cs291a. You will find there
useful information such as announcements, deadlines, syllabus, assignments, and on-line copies of relevant research papers. It is your
responsibility to check the Web site on a regular basis.
It is never too early to start thinking about what you want to do for the class project. As mentioned, considerable freedom will be given here. An in-depth survey of a particular topic can be appropriate If some implementation, testing, and comparing study is a part (i.e., some programming exercise is included). A programming project should involve validation on real data sets (best if it is related to your own research so your effort is not wasted). As most classifiers are fairly straightforward to implement, and those complicated ones, such as neural network classifiers, trees, and SVM, are available publicly (with or without the source codes), e.g., Matlab Neural Network Toolbox, sklearn and Pandas. You can also find public data archives at many web sites, such as PR Archives and UCI Machine Learning web site. The complexity of the project most likely will be in data collection, feature selection, performance tuning, cross validation, comparison of performance using different classifiers, and validation and interpretation of results. For graduate students, particular emphasis will be on the creativity and originality of the project. You need to let me know as soon as you are done with the second assignment and no later than the 8th week what your project idea is.
No elaborate project proposal is needed. Just drop by an email telling me the rough idea of your project (topics, data sets, tools, techniques, partners, etc.) This could be a single-person effort or a group project. You need to get my approval if you want to work as a team. More substantial efforts in research, data collection, implementation, validation, and creativity are expected for team projects.