Apache DataFu
Blog
Apache DataFu™
Getting Started
DataFu Spark
DataFu Pig
DataFu Hourglass
DataFu Spark Docs
Guide
Scaladocs
DataFu Pig Docs
Guide
Javadocs
DataFu Hourglass Docs
Concepts
Javadocs
Community
Mailing Lists
Contributing
Issues
Wiki
Apache Software Foundation
Homepage
License
Sponsorship
Security
Thanks
Apache DataFu
Apache DataFu™ is a collection of libraries for working with large-scale data in Hadoop.
The project was inspired by the need for stable, well-tested libraries for data mining and statistics.
It consists of three libraries:
Apache DataFu Spark
: a collection of utils and user-defined functions for
Apache Spark
Apache DataFu Pig
: a collection of user-defined functions and macros for
Apache Pig
Apache DataFu Hourglass
: an incremental processing framework for
Apache Hadoop
in MapReduce
To begin using it, see our
page. If you'd like to help contribute, see
Contributing
About the Project
Apache DataFu Spark
Apache DataFu Spark is a collection of utils and user-defined functions for
Apache Spark
This library is based on an internal PayPal project and was open sourced in 2019. It has been used by production workflows at PayPal since 2017.
All of the codes is unit tested to ensure quality.
Check out the
Getting Started
guide to learn more.
Apache DataFu Pig
Apache DataFu Pig is a collection of useful user-defined functions for data analysis in
Apache Pig
This library was open sourced in 2010 and continues to receive contributions, having reached 1.0
in September, 2013. It has been used by production workflows at LinkedIn since 2010.
It is also included in Cloudera's
CDH
and
Apache Bigtop
. All of the UDFs are unit tested to ensure quality.
Check out the
Getting Started
guide to learn more.
Apache DataFu Hourglass
Apache DataFu Hourglass is a library for incrementally processing data using Hadoop MapReduce.
This library was inspired by the prevalance of sliding window computations over daily tracking
data at LinkedIn. Computations such as these typically happen at regular intervals (e.g. daily, weekly),
and therefore the sliding nature of the computations means that much of the work is unnecessarily repeated.
DataFu's Hourglass was created to make these computations more efficient, yielding sometimes 50-95% reductions
in computational resources.
Work on this library began in early 2013, which led to a
paper
presented
at
IEEE BigData 2013
. It is currently in production use at LinkedIn.
Check out the
Getting Started
guide to learn more.
Copyright © 2011-2025 The Apache Software Foundation, Licensed under the
Apache License, Version 2.0
Apache DataFu, DataFu, Apache Pig, Apache Hadoop, Hadoop, Apache, and the Apache feather logo are either registered trademarks or trademarks of the
Apache Software Foundation
in the United States and other countries.
US