Big data and beyond
Submitted by Amit Dev (@amitdev) on Tuesday, 29 December 2015
Technical level: Intermediate Status: Submitted
Overview of big data processing using Apache Spark compute model and discuss about social side of big data.
This could be a two-part talk:
1. Initial 80% of the talk is technical. How to process big data with Apache Spark and related tools. Basically a hands on introduction to the Spark compute model (which is simpler than the Hadoop map-reduce model).
2. Last 20% of the talk is non-technical: What are the implications of big data on society? How can FOSS play a role there?
If you’ve heard about the term ‘big data’ you are all set. Idea of Python/Scala might help in understanding the code samples.
I’ve lot of years of industry experience. More importantly, a GNU/Linux user since 1999 and contributor to some open source projects like duckduckgo, python libraries etc.