CS 349: Data Mining, Search, and the World Wide Web
Tuesdays and Thursdays 4:15 - 5:30 in Bldg 370, Room 370 on the Main Quad
Instructors: Sergey Brin and Lawrence Page
Tues and Thurs 5:30 - 7:00 or by appointment.
firstname.lastname@example.org and email@example.com
Course Assistant: Diane Tang
Gates 416: Mon - Wed 11:15 - 12:15 or by appointment.
Over the past two years there has been a close collaboration between the Data Mining Group (MIDAS) and the
Digital Libraries Group at Stanford in the area of Web research. It has culminated in the WebBase project
whose aims are to maintain a local copy of the World Wide Web (or at least a substantial portion thereof) and to
use it as a research tool for information retrieval, data mining, and other applications. This has led to the
development of the PageRank algorithm, the Google search engine, the DIPRE algorithm, and a number of
other works which represent the cutting edge of research on the Web today (see WebBase Publications).
The topics of this class are data mining and information retrieval in the context of the World Wide Web. First,
we will cover background material in data mining and information retrieval that is relevant to the class. Second,
we will cover recent advances made at Stanford (PageRank, DIPRE,...) and elsewhere (Kleinberg, Mitchell,...).
Third and most important students will get the opportunity to work hands on with the WebBase as this will be a
project class. We have already modularized a large part of the code to give people the opportunity to work with
it and will continue to do so throughout the summer. Several people have already taken advantage of the code.
The current WebBase repository consists of roughly 25 million web pages amounting to 150 GB of HTML.
- A strong knowledge of C.
- Working knowledge of C++.
- Very basic statistics, graph theory and linear algebra.
Very Tentative Syllabus
- Introduction: 1
- Data Mining: 5
Publications of IBM's QUEST project
- 10/1 Market Basket (slides)
R. Agrawal, T. Imielinski, A. Swami:
``Mining Associations between Sets of Items in Massive Databases'',
Proc. of the ACM SIGMOD Int'l Conference on Management of Data,
Washington D.C., May 1993, 207-216.
Dynamic Itemset Counting and Implication Rules for
Market Basket Data
by Sergey Brin. Rajeev Motwani, Jeffrey D. Ullman and Shalom Tsur.
We present and algorithm for counting large itemsets faster than
previous algorithms. We rely on partial results to guide the mining
Proceedings of the ACM SIGMOD International Conference on Management of Data,
pp. 255-264, Tuscon, Arizona, May 13-15 1997. (html
, postscript, gzipped ps,
- 10/6 Causality
Scalable Techniques for Mining Causal Structures
by C. Silverstein, S. Brin, R. Motwani, and J. Ullman. VLDB '98.
- 10/8 WebBase 2
- 10/13 Classification and Singular Value Decomposition (slides - html postscript)
SGI's MLC++ Library
- 10/15 Clustering Techniques (slides - html postscript)
Berkeley Clustering Demo
- *** Project Proposals Due ***
- 10/20 Data Mining in the Real World
- Search: 3
- 10/22 Standard IR
- 10/27 New Technologies
- 10/29 Latent Semantic Indexing
Bellcore's LSI site
- 11/3 WebBase 3
- *** Milestone Due ***
- Web: 6
- 11/5 Search Engines 1 - basics, size, evaluation
- 11/10 Search Engines 2 - crawling, robots.txt, ...
- 11/12 PageRank, Kleinberg
- 11/17 DIPRE
- 11/19 DEC Research
- 11/24 Classification of Web Pages
- *** Final Project Due ***
Last modified: Sat Oct 24 23:18:37 PDT 1998