WAC Workshop at Stanford
In June 2012 we held an archiving workshop at Stanford University.
A number of students and faculty attended from Old Dominion and Harding
Universities. Frank McCown of Harding organized the event.
Special about this workshop was that it brought together different disciplines,
like library science, law, and computer science.
See slides from the presentations by
and Aaron Binns
We heard about archiving efforts in a number of institutions, both
within and outside of the Web Archive Cooperative. For many of the
students this workshop was their first visit to Stanford, and their
first exposure to events of this type.
Quotes from participants:
"This was so exciting!"
"This was a semester's worth of material in one day."
"I got so many new ideas today."
"It was great to observe how the senior scientists questioned each other to learn."
In addition to energizing and inspiring all participants, the workshop
was an opportunity for the project principals to coordinate. We consequently
decided to develop conversion technology from the 280TB Stanford
WebBase holdings to the Web Archiving (WARC) format. This format is
used at the Internet Archive, as well as the Library of Congress and
the California Digital Library. The Old Dominion Memento project builds
on the standard, as do a number of tools, like the Wayback Machine. We
therefore expect that this development effort will serve to unify
some significant holdings.
Post Workshop Technical Activities
Our WAC workshop in June 2012 kicked off a number of technical efforts.
Our decision to unify our formatting in support of the WARC standard was
crawls may now be requested via a Web interface in
either their original formats, or as WARC formatted packages.
WAC investigator Michael Nelson's presentation at the workshop stressed
the importance of projects that make Web archives useful today
as opposed to merely promising payback years from now. Such products,
the arguments goes, will ensure enduring interest in archiving activities.
In pursuit of this goal our Stanford WAC contingent embarked on an unusual
employment of archives: EchoTrees
. This small, ongoing, exploration
attempts to employ archives to support communications of a mute quadriplegic
collaborator with conversation partners. This collaborator,
, types communications on his laptop using a head
tracker. For his conversation partners, the wait for answers during
Henry's conversation turns is passivating and discouraging. Our experimental
solution is to use varying Web and other collections to predict
multiple phrase outcomes from any word that Henry types. These outcomes
are visualized in a word tree, as exemplified in the following figure from
our technical report
EchoTree: Engaged Conversation when Capabilities are Limited
Collocated or remote conversation partners can see these visualizations
on their Web browsers. They can manipulate the trees, and discuss possibilities
with each other. Most importantly, conversation partners propose to Henry some
phrase outcome he might have in mind, saving him typing time. We are in
the process of evaluating various aspects of this design.
Panel Discussion at Infolab Workshop
In April, 2013 during our
the WAC project organized a panel
Data in the Cloud: For What and For Whom?
The workshop was organized this time as a collaboration of universities at Stanford, UC Berkeley, UC Santa Cruz,
UC Merced, Cal Poly, and Davis.
The workshop was attended by some 200 visitors from those universities, as well as
surrounding Silicon Valley industry. The panel stimulated a discussion on implications
of moving large segments of personal information into the cloud, which is generally
owned by private interests. We heard from panelist Brewster Kahle
about the Internet
Archive having been served one of the controversial "National Security Letters."
of the Stanford Center for Internet and Society summarized
for the audience legislative developments around information privacy
and digital rights
of Dezine provided examples for creative outcomes made
possible by contributions of information to the 'commons.'
On the sidelines of this event it was decided to donate a snapshot of Stanford's
WebBase to the Internet Archive. This effort will involve conversion of the
280TB (uncompressed) collection into the WARC format, as well as transfer and integration
into the Internet Archive collection. We will use our 60 node compute cluster to
accomplish this task.
Static Project Information
Challenge: Describing Resources
Each federation member has a
set of ``resources'', e.g., web crawls, query logs, crawling software,
etc. To be usable, each resource needs to be described in a way that
can be understood for other federation members. How was the resource
obtained? On what dates? What does it contain? Who can access the
resource? Who do we ``compare'' archives and their holdings? While
standards are emerging, their resource descriptions are not yet
detailed enough to allow integration with other resources. The
challenge is to identify descriptions that truly facilitate
experimentation and integration, and at the same time are reasonable
for the resource owner to generate.
Challenge: Resource Discovery and Characterization.
Thrust 1 (Resource Description): Create a comprehensive ``taxonomy'' of descriptive
features and access mechanisms (APIs) for a wide range of resource types.
Develop a cost/benefit ratio for each feature/mechanism that
describes how difficult it is to obtain/implement the feature/mechanism, and how useful it is
to support experimentation and research.
Identify, promote and develop metrics for quantifying and comparing
Study how to include new features/mechanisms in existing or new standards.
Build a reference implementation for an archive
that supports advanced resource description/access
for a variety of resources.
A WAC needs a discovery service that lets researchers find resources of interest.
Resource owners can manually register resources
at the discovery service, or the service can automatically
harvest information about emerging resources
(e.g., by monitoring crawler traffic at Web sites).
If a resource is not fully described, the discovery service
may be able to analyze the resource and extract
its characteristics (e.g., site depth of a crawl, coverage,
Challenge: Linking and Combining Resources.
Thrust 2 (Discovery)
Study and evaluate options for a resource discovery service.
We will explore three goals for such a service:
(1) the manual or automated discovery of entire existing Web related
archives; (2) the selection
among known archives of the ones that
support a specific research question; and (3)
the identification of individual resources from within the
We will also develop tools for characterizing discovered archives,
especially for the case where the archive does not provide rich
Characterization of an archive includes elements such as an estimate
of the archive's coverage, particulars of the crawling parameters,
like dates/frequencies, crawl duration, depth, per-site ceiling on the
number of collected pages, content statistics, and link structure.
Using open-source software as much as possible,
we will build, operate, and evaluate a discovery service for our WAC.
Finally, we will also support what we call forward discovery,
i.e., the identification of candidates for future archiving.
For WAC forward discovery we will provide a clearinghouse
where the community can express recommendations.
Such a clearinghouse is needed, because
recommending parties do not often themselves possess archiving
capacity. On the other hand, the community at large is an
indispensable resource for identifying niches of interest on the Web
that might be of importance in the future.
The WAC provides integrated access to independent resources. This
integration requires sophisticated resource and metadata translation
mechanisms. For instance, URLs in one archive need to be mapped to
ones in another; annotation tags in one resource need to be translated
to their synonyms in another. Redundant (or approximately redundant)
objects need to be identified, merged and possibly exploited (e.g., if
an archived URI is damaged, are there redundant or similar URIs that
can be substituted?). Inconsistencies in the way
resources were gathered need to be resolved, or at least described.
For example, how do we unify two Web page crawls, one that visited
sites every 3 days and another that visited sites every 5 days?
Challenge: Preserving Resources.
Thrust 3 (Archive Linking)
We will develop mechnisms for integrating diverse archives,
and will apply the mechnisms to site reconstruction
(from various archives) and archive views (a logical fusion
of resources from multiple sources).
Since integration issues are so challenging, we will
set up an experimental testbed with small but diverse resources.
The testbed will contain several crawls of the same target sites,
each obtained with different crawlers and using different parameters.
The testbed will also contain related resources, e.g., the tags
at Delicious for the same set of sites.
The testbed will let us study and quantify differences among
the crawls, and will let us evaluate strategies for combining
and linking resources.
The WAC preserves past Web states, but who preserves the
WAC content itself? In other words, WAC resources
stored at member archives can be lost due to
hardware failures or the member archive going out of business.
Resources can be preserved through replication, but
(a) member archives must be willing to store backup copies;
(b) the number of desired copies and their location must be determined, and
(c) update propagation mechanisms must be in place
to keep replicas synchronized.
The size and rate of change of WAC resources make
all these aspects especially challenging.
Challenge: Filling the Gaps.
Thrust 4 (Preservation)
We will explore storage trading schemes that allow members
to trade local backup space for remote space.
We will extend the notion of self-preserving objects
to develop a Web archive replication tool.
We will study alternatives for replica synchronization.
For example, is it best to have a single crawler generate
one Web archive that is replicated at two other sites,
or is it better to have three coordinated crawlers that
each create their own archive of the same target sites?
As we conduct our research, we are bound to see gaps in coverage:
data sets that researchers need but are not available anywhere,
or tools that researchers need but have not been developed.
Gaps occur when resources exist but are not shared
(e.g., query logs are often considered sensitive),
or for emerging applications where data collection
tools have not been developed
(e.g., the next Facebook or Twitter-like system).
Challenge: Community Building.
Thrust 5 (Filling Gaps):We will study ways to fill the gaps. For example, it is
possible to gather query logs in a distributed fashion by using the
referrer field of HTTP requests from search engines. Thus, a
community of Web sites can gather a query log, filling an important
gap that exists today. We will also build data gathering tools for
emerging applications (e.g., an archive of Tweeter feeds and
profiles). Another gap exists for what is called the Deep
Web, i.e., information that resides in
backend databases but is displayed through dynamic Web pages. Such
information is very valuable but is hard to find in open Web archives
In addition to filling information gaps, we will also address gaps in
tools that facilitate archiving and resource sharing. For example, we
will adapt annoymization techniques explored by the security community
to the context of Web content, thus making owners more willing to
share their resources.
The success of a WAC will depend on the willingness of members
to gather, implement, and share resources.
In turn, this willingness will depend on
the availability of useful standards and tools,
on the initial seeding of the WAC with a substantial
number of resources, and an understanding
of the legal and social issues related to research
of shared Web resources.
Thrust 6 (Community Building)
We will organize a number of workshops to bring together
key Web Science researchers, to discuss available resources
and impediments to sharing.
These workshops will drive our research,
identifying needed tools and protocols.
With small groups of participants, we will
establish challenge problems to attack,
e.g., combining a set of Web archives.
With restricted participation, we expect
to get access to more resources, demonstrating
that a collective effort can yield benefits to all.
Reports of these results at future workshops
can incentivize others to participate in the WAC.
In addition, we will set up an Advisory Board
of industrial, government, and academic experts
to guide our project.
To keep Web Science vibrant, future researchers and practitioners need
to be trained.
However, current knowledge
(e.g., how to effectively run massive Web crawls,
how to extract meaningful information from massive Web data sets)
is widely dispersed, and current tools are poorly documented.
Thrust 7 (Education)
We will run a Summer Institute for Web Science graduate students.
At this Institute, students will learn to use the latest tools,
and will learn from each other's experiences in dealing with Web data.
In addition, we will develop a one-day workshop which can be offered
at Web Science conferences (WWW, SIGIR, etc.) to educate participants
about the resource made available by the WAC. We will also develop
an undergraduate Web Sciences track for computer science majors that
will use WAC tools.
Project Advisory Board
Martha Anderson, LOC
Pamela Anderson, Berkeley
Christine Borgman UCLA
Patricia Cruse, Cal. Digital Library
Richard Furuta Texas A&M
Alon Halevy, Google
Carl Lagoze, Cornell
Gary Marchionini, U.North Carolina
Raghu Ramakrishnan, Yahoo
Herbert van de Sompel, LANL
Auxiliary Information from Kickoff Meeting
From Alex Thurman
Web Collection Curator
Columbia University Libraries
535 W. 114th Street
New York, NY 10027
Here are some links to resources mentioned during the Workshop.
Columbia is currently surveying three user groups to help guide the design of our web archives access portal. The groups are: human rights researchers (students, faculty); content providers (NGOs whose sites we're archiving); and librarians/archivists. The surveys vary slightly for the 3 different groups, but are largely identical. If you'd like to see and/or complete the survey sent to librarians/archivists, the link follows. When we have our results I can share them with this group if desired.
Take the survey: http://www.surveymonkey.com/s/columbiawebarchives_L
More detailed web archives user studies are available from the Portuguese Web Archive at:
The results of the survey of web archiving initiatives that they conducted and posted on Wikipedia is at:
Two access portals that came up in discussion were the UK Web Archive and Trove
Directory of Existing Archives
As a first order of business we compiled a list of the Web archives that we are aware of. This list is available as a Google Docs spreadsheet. We invite the public to add entries for other archives as they become available. The current list comprises over 1500 entries.
JCDL Archiving Workshop
In the context of the Joint Conference on Digital Libraries (JCDL 2011) this project will organize a workshop
. The meeting will bring together interested parties from major archives, government, private, and academic. We will report on results.
We are working with the distributed computing infrastructure Hadoop. The goal is seamlessly to stream our WebBase archive through a compute cluster for analysis and processing. In this context we contributed an Excel load and store module to the Apache Pig open source project.
Please visit our WebBase archive
, where we make several years of archived Web content available.