RAMP: A Hadoop Extension for Provenance Support

Debugging MapReduce workflows can be a difficult task: their execution is batch-oriented and, once completed, leaves only the data sets themselves to help in the debugging process. Data provenance, which captures how data elements are processed through the workflow, can aid in debugging by enabling backward tracing: finding the input subsets that contributed to a given output element. For example, erroneous input elements or processing functions may be discovered by backward-tracing suspicious output elements. Provenance and backward tracing also can be useful for drilling-down to learn more about interesting or unusual output elements.

RAMP (Reduce And Map Provenance) is an extension to Hadoop that supports provenance capture and tracing for MapReduce workflows. RAMP captures fine-grained provenance by wrapping the RecordReader, Mapper, Combiner, Reducer, and RecordWriter. This wrapper-based approach is transparent to Hadoop, retaining Hadoop’s parallel execution and fault tolerance. Furthermore, in many cases users need not be aware of provenance capture while writing MapReduce jobs–wrapping is automatic, and RAMP stores provenance separately from the input and output data. Our performance experiments show that RAMP imposes reasonable time and space overhead during provenance capture and enables efficient backward tracing without requiring special indexing of provenance information.

Here are our demonstration proposal, presentation slides, and research paper describing this system.

• RAMP: A System for Capturing and Tracing Provenance in MapReduce Workflows.
  H. Park, R. Ikeda, and J. Widom.
  37th International Conference on Very Large Data Bases (VLDB) (Demonstration Track), Seattle, Washington, August 2011.
• RAMP: A System for Capturing and Tracing Provenance in MapReduce Workflows.
  H. Park, R. Ikeda, and J. Widom.
  Fourth Annual Hadoop Summit (Applications and Research Track), Santa Clara, California, June 2011.
• Provenance for Generalized Map and Reduce Workflows.
  R. Ikeda, H. Park, and J. Widom.
  Fifth Biennial Conference on Innovative Data Systems Research (CIDR), Pacific Grove, California, January 2011.

The source code is available at github.com/hyunjung/hadoop-common and github.com/hyunjung/pig.

Example: Temporal Query Phrase Popularity from Pig Tutorial