Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Michael Armbrust

“As Apache Spark becomes more widely adopted, we have focused on creating higher-level APIs that provide increased opportunities for automatic optimization. In this talk, I give an overview of some of the exciting new API’s available in Spark 2.0, namely Datasets and Structured Streaming.  Together, these APIs are bringing the power of Catalyst, Spark SQL's query optimizer, to all users of Spark.  I'll focus on specific examples of how developers can build their analyses more quickly and efficiently simply by providing Spark with more information about what they are trying to accomplish.” - Michael

Slides: http://www.slideshare.net/databricks/...

Databricks Blog:  "Deep Dive into Spark SQL’s Catalyst Optimizer"
https://databricks.com/blog/2015/04/1...

// About the Presenter //
Michael Armbrust is the lead developer of the Spark SQL project at Databricks. He received his PhD from UC Berkeley in 2013, and was advised by Michael Franklin, David Patterson, and Armando Fox. His thesis focused on building systems that allow developers to rapidly build scalable interactive applications, and specifically defined the notion of scale independence. His interests broadly include distributed systems, large-scale structured storage and query optimization.

Follow Michael on -
Twitter: Twitter: michaelarmbrust
LinkedIn: LinkedIn: michaelarmbrust

• #science_technology
• #spark_sql
• #sparksql
• #michael_armbrust
• #spark_dataframes
• #spark_20
• #apache_spark
• #structuring_spark
• #continuous_applications