Fine-grained and efficient lineage querying of collection-based workflow provenance
Authors
- Paolo Missier (University of Manchester, UK)
- Norman W. Paton (University of Manchester, UK)
- Khalid Belhajjame (University of Manchester, UK)
Abstract
The management and querying of workflow provenance data underpins a collection of activities, including the analysis of workflow results, and the debugging of workflows or services. Such activities require efficient evaluation of lineage queries over potentially complex and voluminous provenance logs. Naive implementations of lineage queries navigate provenance logs by joining tables that represent the flow of data between connected processors invoked from workflows. In this paper we provide an approach to provenance querying that: (i) avoids joins over provenance logs by using information about the workflow definition to inform the construction of queries that directly target relevant lineage results; (ii) provides fine grained provenance querying, even for workflows that create and consume collections; and (iii) scales effectively to address complex workflows, workflows with large intermediate data sets, and queries over multiple workflows.
Session
EDBT Research Session 9: Data Provenance (Thursday, March 25, 14:00—15:30)