Web Archive Cooperative
Making Web Archives Useful Today

Supported by the National Science Foundation (1009916)

The Web Archive Cooperative (WAC) project aims to advance technology and practices for the non-commercial archiving of theWeb. The effort includes principle investigators from Old Dominion University, Harding University, and Stanford University. You are currently visiting the Stanford project portion. Please also visit the Old Dominion and Harding arms of the project.

Please see below for a summary of challenges we set ourselves. You will also find a list of our advisory board. We are very fortunate to have these seasoned researchers guide us along the way.

Towards the top of this page (that is right below) you will find project accomplishments as they evolve. Below these news items you find more static information about our project.



Project Updates

WebBase Crawling Activity Comes to an End

Sadly, as of Oct 31, 2013 funding will run dry for continued update of our WebBase. Since 2005 we have crawled the same 40,000 sites every single month, creating a unique time series. Every three months we collected all of the US Federal Government .gov pages, drilling down through US States to townships every six months. This collection spans major events, like the change of US administration from Bush/Cheney to Obama/Biden, a number of other electoral events, triumphs, as well as tragedies. As WebBase curation stressed not breadth, but consistency in sampling times, the collection is unlike any other. Provided we can preserve these holdings, they will constitute a unique resource for political science, sociology, and other disciplines. This unfortunate discontinuation illustrates how tenuous Web archiving efforts still remain today.

Advisory Board Meeting

In September, 2010 we kicked the project off by inviting our advisory board to Stanford. We presented our plans, and listened to the board's suggestions. We received valuable pointers to efforts elsewhere, both US and international. These leads later exposed significant differences among US and many European national collection efforts. It seems that while European government-run Web archiving efforts are broad, and probably quite complete, the resulting archives are often closed for all practical purposes. For example, several countries limit access to a handful of terminals in their national library buildings.Kickoff

Global Web Archiving Workshop at JCDL

On June 17 and 18, 2011 we organized a workhop Web Archive Globalization in the context of the Joint Conference on Digital Libraries. The roughly 20 participants had lively discussions in response to four prepared presentations. These talks were given by Eric Hetzner (California Digital Library), Nicholas Taylor (Library of Congress), Brad Tofel (Internet Archive), and Rob Sanderson (Los Alamos National Lab). The Library of Congress slides are available online, as are the presentations by the California Digital Library, the Internet Archive, and those of the Los Alamos National Lab.

The attending PIs of the Web Archive Cooperative project met ahead of the workshop to coordinate both the workshop and the project in general.
WAC Coordination Meeting JCDL 2011 Group Photo Dinner side1 Dinner side2. Paper underwear
More discussion was had over dinner. A side note: free travel advice. Light and small five packs of paper underwear and socks (you had to be there to see the beauty...). We will add additional workshop notes shortly.

Web Archiving Receives Publicity

Mike Nelson's part of the project enjoyed a round of good publicity around web archiving, which started with an Old Dominion Web Science and Digital Libraries Research Group blog update post. That entry was picked up by the Chronicle of Higher Education with an article on July 6, 2011. That story in turn led to an article in the Washington Post on July 17, 2011, which then elicited a short TV interview on "Canada AM" on July 21.

Project Introduced to Low-Income High-Achievement High-School Students

With funding from Google, Stanford University's Computer Science Deparment organized a series of on-campus events for high achieving, low-income high school students of color from across the country. The students were selected for their passion around math and science. The goal is to inspire the students towards an engineering career. During the one-week residential and academic program, students take rigorous coursework that prepares them to excel in science, technology, engineering or mathematics, with a strong preference for Computer Science. The WAC project PI Prof. Hector Garcia-Molina introduced the project and other research examples to a group of fouteen Mexican Americans/Latinos and sixteen African Americans. Many of the students were first-generation citizens.
							 Prof. Garcia-Molina
							 speaking LEAD program summer
						  students(1) LEAD students(2)

Web Archive Cooperative Volunteers at Hack-the-Future

The WAC team volunteered at a Bay Area event 'Hack-the-Future,' in which children engage in technology projects designed to inspire them towards careers in the sciences. Andreas Paepcke of the WAC team developed a very simple method for programming robot interactions (separate funding). This Hack-the-Future project featured an implementation of this method created by engineers at Willow Garage, who provided the robot for the event.
Girl in robot
						    embrace Help with

This girl's program works...mostly.

Web Archive Workshop Coming Up

Preparations are on the way for our June 2012 workshop at Stanford. Harding University is organizing. Details here.

WAC Team Volunteers at San Jose Tech Museum: CS Outreach to Kids

In an effort to induce interest in Computer Science at an early age, members of the Web Archive Cooperative volunteered at the San Jose Tech Museum. Small girl with
robot Adults and kids Girlscouts
     and robot Girls at computers Children as well as adults grew deeply involved in programming a large robot to gesture and dance. One of the WAC members wrote the prototype for the underlying software (under separate funding). The system was exhibited on six consecutive weekends at the museum, with help from us volunteers. The age range of those attracted to the exhibit, and truly engrossed in it was from 2 years to full adulthood, an astounding spread.

Stanford WAC Team at Stanford Datafest

On May 19, 2012, the Stanford arm of the Web Archive Cooperative project participated at the Stanford Datafest. The event was organized as part of the Computational Reporting program. Participants consisted of journalists, political scientists, and computing people. Participants formed teams, which selected topics concerning campaign finance. A representative of the Sunshine Foundation was on hand to consult about available data sources and promising search strategies.

As projected by our WAC NSF proposal, social sciences are increasingly aware that mining the Web and its archives can yield important insights. Groups began with plans around a whiteboard. Once groups had decided on topics to investigate, small and large teams formed to pursue answers to their questions.

Small group at
						 Datafest. Large group at
						 Datafest. Multiple groups
						     at Datafest.

WAC Member Honored: Knight Fellow 'Favorite Professor'

In the context of the WAC project Andreas collaborated with one of the 2011/2012 Stanford Knight Fellow journalists, Liz McClure. In return, Liz nominated Andreas as her 'Favorite Professor' during her stay at Stanford. A reception brought together the outgoing Knight Fellows and 'their professors.'
Andreas with Knight
					      Fellow and Faculty Group photo Group Stanford
     Faculty and Knight Fellows.

WAC Workshop at Stanford

In June 2012 we held an archiving workshop at Stanford University. A number of students and faculty attended from Old Dominion and Harding Universities. Frank McCown of Harding organized the event. Special about this workshop was that it brought together different disciplines, like library science, law, and computer science. See slides from the presentations by Eric Hetzner and Aaron Binns.
					      at WAC Workshop 2012
We heard about archiving efforts in a number of institutions, both within and outside of the Web Archive Cooperative. For many of the students this workshop was their first visit to Stanford, and their first exposure to events of this type.
Cathy Marshall
					      at WAC Workshop 2012 Participants
					      at WAC Workshop 2012
Quotes from participants:

"This was so exciting!"
"This was a semester's worth of material in one day."
"I got so many new ideas today."
"It was great to observe how the senior scientists questioned each other to learn."

In addition to energizing and inspiring all participants, the workshop was an opportunity for the project principals to coordinate. We consequently decided to develop conversion technology from the 280TB Stanford WebBase holdings to the Web Archiving (WARC) format. This format is used at the Internet Archive, as well as the Library of Congress and the California Digital Library. The Old Dominion Memento project builds on the standard, as do a number of tools, like the Wayback Machine. We therefore expect that this development effort will serve to unify some significant holdings.

Post Workshop Technical Activities

Our WAC workshop in June 2012 kicked off a number of technical efforts. Our decision to unify our formatting in support of the WARC standard was realized. WebBase crawls may now be requested via a Web interface in either their original formats, or as WARC formatted packages. WAC investigator Michael Nelson's presentation at the workshop stressed the importance of projects that make Web archives useful today, as opposed to merely promising payback years from now. Such products, the arguments goes, will ensure enduring interest in archiving activities. In pursuit of this goal our Stanford WAC contingent embarked on an unusual employment of archives: EchoTrees. This small, ongoing, exploration attempts to employ archives to support communications of a mute quadriplegic collaborator with conversation partners. This collaborator, Henry Evans, types communications on his laptop using a head tracker. For his conversation partners, the wait for answers during Henry's conversation turns is passivating and discouraging. Our experimental solution is to use varying Web and other collections to predict multiple phrase outcomes from any word that Henry types. These outcomes are visualized in a word tree, as exemplified in the following figure from our technical report EchoTree: Engaged Conversation when Capabilities are Limited:
EchoTree phrase prediction visualization.

Collocated or remote conversation partners can see these visualizations on their Web browsers. They can manipulate the trees, and discuss possibilities with each other. Most importantly, conversation partners propose to Henry some phrase outcome he might have in mind, saving him typing time. We are in the process of evaluating various aspects of this design.

Panel Discussion at Infolab Workshop

In April, 2013 during our biennial Infolab-wide workshop the WAC project organized a panel Data in the Cloud: For What and For Whom? The workshop was organized this time as a collaboration of universities at Stanford, UC Berkeley, UC Santa Cruz, UC Merced, Cal Poly, and Davis.
The workshop was attended by some 200 visitors from those universities, as well as surrounding Silicon Valley industry. The panel stimulated a discussion on implications of moving large segments of personal information into the cloud, which is generally owned by private interests. We heard from panelist Brewster Kahle about the Internet Archive having been served one of the controversial "National Security Letters." Jennifer Granick of the Stanford Center for Internet and Society summarized for the audience legislative developments around information privacy and digital rights management. Shaukat Shamim of Dezine provided examples for creative outcomes made possible by contributions of information to the 'commons.' On the sidelines of this event it was decided to donate a snapshot of Stanford's WebBase to the Internet Archive. This effort will involve conversion of the 280TB (uncompressed) collection into the WARC format, as well as transfer and integration into the Internet Archive collection. We will use our 60 node compute cluster to accomplish this task.

Static Project Information


Challenge: Describing Resources
Each federation member has a set of ``resources'', e.g., web crawls, query logs, crawling software, etc. To be usable, each resource needs to be described in a way that can be understood for other federation members. How was the resource obtained? On what dates? What does it contain? Who can access the resource? Who do we ``compare'' archives and their holdings? While standards are emerging, their resource descriptions are not yet detailed enough to allow integration with other resources. The challenge is to identify descriptions that truly facilitate experimentation and integration, and at the same time are reasonable for the resource owner to generate.

Challenge: Resource Discovery and Characterization.
A WAC needs a discovery service that lets researchers find resources of interest. Resource owners can manually register resources at the discovery service, or the service can automatically harvest information about emerging resources (e.g., by monitoring crawler traffic at Web sites). If a resource is not fully described, the discovery service may be able to analyze the resource and extract its characteristics (e.g., site depth of a crawl, coverage, diameter). Challenge: Linking and Combining Resources.
The WAC provides integrated access to independent resources. This integration requires sophisticated resource and metadata translation mechanisms. For instance, URLs in one archive need to be mapped to ones in another; annotation tags in one resource need to be translated to their synonyms in another. Redundant (or approximately redundant) objects need to be identified, merged and possibly exploited (e.g., if an archived URI is damaged, are there redundant or similar URIs that can be substituted?). Inconsistencies in the way resources were gathered need to be resolved, or at least described. For example, how do we unify two Web page crawls, one that visited sites every 3 days and another that visited sites every 5 days? Challenge: Preserving Resources.
The WAC preserves past Web states, but who preserves the WAC content itself? In other words, WAC resources stored at member archives can be lost due to hardware failures or the member archive going out of business. Resources can be preserved through replication, but (a) member archives must be willing to store backup copies; (b) the number of desired copies and their location must be determined, and (c) update propagation mechanisms must be in place to keep replicas synchronized. The size and rate of change of WAC resources make all these aspects especially challenging. Challenge: Filling the Gaps.
As we conduct our research, we are bound to see gaps in coverage: data sets that researchers need but are not available anywhere, or tools that researchers need but have not been developed. Gaps occur when resources exist but are not shared (e.g., query logs are often considered sensitive), or for emerging applications where data collection tools have not been developed (e.g., the next Facebook or Twitter-like system). Challenge: Community Building.
The success of a WAC will depend on the willingness of members to gather, implement, and share resources. In turn, this willingness will depend on the availability of useful standards and tools, on the initial seeding of the WAC with a substantial number of resources, and an understanding of the legal and social issues related to research of shared Web resources. Challenge: Education.
To keep Web Science vibrant, future researchers and practitioners need to be trained. However, current knowledge (e.g., how to effectively run massive Web crawls, how to extract meaningful information from massive Web data sets) is widely dispersed, and current tools are poorly documented.

Project Advisory Board

Martha Anderson, LOC
Pamela Anderson, Berkeley
Christine Borgman UCLA
Patricia Cruse, Cal. Digital Library
Richard Furuta Texas A&M
Alon Halevy, Google
Carl Lagoze, Cornell
Gary Marchionini, U.North Carolina
Raghu Ramakrishnan, Yahoo
Herbert van de Sompel, LANL


Auxiliary Information from Kickoff Meeting

From Alex Thurman
Web Collection Curator
Columbia University Libraries
535 W. 114th Street
New York, NY 10027
Here are some links to resources mentioned during the Workshop.

Columbia is currently surveying three user groups to help guide the design of our web archives access portal. The groups are: human rights researchers (students, faculty); content providers (NGOs whose sites we're archiving); and librarians/archivists. The surveys vary slightly for the 3 different groups, but are largely identical. If you'd like to see and/or complete the survey sent to librarians/archivists, the link follows. When we have our results I can share them with this group if desired.

Take the survey: http://www.surveymonkey.com/s/columbiawebarchives_L

More detailed web archives user studies are available from the Portuguese Web Archive at: http://sobre.arquivo.pt/about-the-archive/publications

The results of the survey of web archiving initiatives that they conducted and posted on Wikipedia is at: http://en.wikipedia.org/wiki/List_of_Web_Archiving_Initiatives

Two access portals that came up in discussion were the UK Web Archive and Trove

Directory of Existing Archives

As a first order of business we compiled a list of the Web archives that we are aware of. This list is available as a Google Docs spreadsheet. We invite the public to add entries for other archives as they become available. The current list comprises over 1500 entries.

JCDL Archiving Workshop

In the context of the Joint Conference on Digital Libraries (JCDL 2011) this project will organize a workshop. The meeting will bring together interested parties from major archives, government, private, and academic. We will report on results.

Software Releases

We are working with the distributed computing infrastructure Hadoop. The goal is seamlessly to stream our WebBase archive through a compute cluster for analysis and processing. In this context we contributed an Excel load and store module to the Apache Pig open source project.

Data Access

Please visit our WebBase archive, where we make several years of archived Web content available.